Parsing JSON files

Here you can suggest additions and modifications to BB4W or BBCSDL
Mijndert Rooth
Posts: 5
Joined: Thu 23 Jul 2020, 14:38

Parsing JSON files

Post by Mijndert Rooth »

Is there somebody who has knowledge how to parse JSON files?

I have a solar invertor which will answer on API calls with a textfile in JSON format.
I call de API from within a "BBC BASIC for Windows" program and would like to analyse the returned data and present selected data in a window.

The XMLLIB library provides a number of functions to simplify the parsing of XML files.
I am looking for comparable functions for parsing JSON files.

I welcome every suggestion.
Mijndert.


P.S. Being Dutch undoubtedly has a negative effect on my English, my appologies.
Acorn BBC Micro, model B with 6502 Second Processor.

RichardRussell
Posts: 386
Joined: Tue 15 Oct 2019, 09:10

Re: Parsing JSON files

Post by RichardRussell »

Mijndert Rooth wrote:
Thu 23 Jul 2020, 15:03
The XMLLIB library provides a number of functions to simplify the parsing of XML files.
I am looking for comparable functions for parsing JSON files.
I don't know anything about JSON files, but doing a web search found this, which claims to convert a JSON file to one or more CSV files. I'm not suggesting that this is necessarily the way to go, but it makes me wonder whether an XMLLIB-like library is the best approach. It seems that JSON files can contain arrays, so maybe what is needed is a library that can extract the contents into one or more BBC BASIC arrays?
If you have a comment about the style or tone of this message please report it to the moderators by clicking the exclamation mark icon, rather than complaining on the public forum.

irrelevant
Posts: 1
Joined: Sat 14 Apr 2018, 17:09

Re: Parsing JSON files

Post by irrelevant »

Would it be possible to post an example of the json format data you wish to decode? Json can encode various types of data, but if you are only using a subset, or it is a simple construct, then it might be easier to write a quick parser to deal with it than port an entire library, most of which word end up not being used.
Last edited by irrelevant on Sat 25 Jul 2020, 23:41, edited 1 time in total.

User avatar
JeremyNicoll
Posts: 2
Joined: Sun 26 Jul 2020, 22:22
Location: Edinburgh

Re: Parsing JSON files

Post by JeremyNicoll »

I have the impression that something like XMLLIB would be needed to do the job properly, for arbitrary JSON data, because - so far as I understand it - JSON structures can be contained recursively inside parent structures, much as is true of XML data.

For example, look at https://en.wikipedia.org/wiki/JSON section 6.3 Samples, where a simple structure is shown in JSON, YAML and XML.

Looking at Richard's XML parsing example at http://www.bbcbasic.co.uk/wiki/doku.php ... ds&s[]=xml where one can only sensibly extract data if eg one knows that <item>s are within <channels> which are within <rss>, I expect that (using the wikipedia example) one needs to know that eg phone numbers are stored inside the structure starting at " "phone numbers": [ ".

There's probably a few places around the web where one can paste in a JSON string and see its contents laid out in a nested fashion. I found one at: http://jsonviewer.stack.hu/ but I'm sure there will be others. Beware as always that if you provide a data string to such a service you'd not want there to be any data confidential to you in that string. I "collapsed" the JSON example from the wikipedia page, giving me one longer string:

{"first name": "John", "last name": "Smith", "age": 25, "address": { "street address": "21 2nd Street", "city": "New York", "state": "NY", "postal code": "10021" }, "phone numbers": [ { "type": "home", "number": "212 555-1234" }, { "type": "fax", "number": "646 555-4567" } ], "sex": { "type": "male" } }

Bear in mind that spaces between terms are not required so the string could also have been

{"first name":"John","last name":"Smith","age":25,"address":{"street address":"21 2nd Street","city":"New York","state":"NY","postal code":"10021"},"phone numbers":[{"type":"home","number":"212 555-1234"},{"type": "fax","number":"646 555-4567"}],"sex":{"type":"male"}}

and pasted that into the viewer (in its "Text" tab) and then clicked its VIewer tab, which shows the corresponding nested structure.


On the other hand if the actual instances of JSON data that the OP sees are simple, it may be possible (eg) to find the n'th occurrence of some device or application-specific literal in the JSON string and then extract (say) the next 100 bytes, and chop that up.

If you look at the list of packages at the foot of: https://www.json.org/json-en.html some of them might give an idea of the sorts of functions a JSON encoder/decoder might need. I looked at just one of those, the PureBasic page at: https://www.purebasic.com/documentation/json/index.html - which makes it look as if it might be complicated. On the other hand, that interface may have defined functions for every represented datatype, and maybe that's more complicated than is absolutely necessary.

RichardRussell
Posts: 386
Joined: Tue 15 Oct 2019, 09:10

Re: Parsing JSON files

Post by RichardRussell »

I understand that at least one more reply to this thread has been posted, but it has seemingly not been approved by the moderator.
If you have a comment about the style or tone of this message please report it to the moderators by clicking the exclamation mark icon, rather than complaining on the public forum.

Mijndert Rooth
Posts: 5
Joined: Thu 23 Jul 2020, 14:38

Re: Parsing JSON files

Post by Mijndert Rooth »

RichardRussell wrote:
Sat 25 Jul 2020, 14:09

I don't know anything about JSON files, but doing a web search found this, which claims to convert a JSON file to one or more CSV files. I'm not suggesting that this is necessarily the way to go, but it makes me wonder whether an XMLLIB-like library is the best approach. It seems that JSON files can contain arrays, so maybe what is needed is a library that can extract the contents into one or more BBC BASIC arrays?
Thanks for your suggestion. I will look into it.
I have found the JSON standard, so in the meanwile I will try to write some procedures and functions on my own.
Last edited by Mijndert Rooth on Tue 28 Jul 2020, 08:17, edited 1 time in total.
Acorn BBC Micro, model B with 6502 Second Processor.

Mijndert Rooth
Posts: 5
Joined: Thu 23 Jul 2020, 14:38

Re: Parsing JSON files

Post by Mijndert Rooth »

irrelevant wrote:
Sat 25 Jul 2020, 23:39
Would it be possible to post an example of the json format data you wish to decode? Json can encode various types of data, but if you are only using a subset, or it is a simple construct, then it might be easier to write a quick parser to deal with it than port an entire library, most of which word end up not being used.
An example:
------------------------------------------------------------------------------
{
"Body" : {
"Data" : {
"DAY_ENERGY" : {
"Unit" : "Wh",
"Values" : {
"1" : 4346
}
},
"PAC" : {
"Unit" : "W",
"Values" : {
"1" : 3105
}
},
"TOTAL_ENERGY" : {
"Unit" : "Wh",
"Values" : {
"1" : 6645459
}
},
"YEAR_ENERGY" : {
"Unit" : "Wh",
"Values" : {
"1" : 3938563
}
}
}
},
"Head" : {
"RequestArguments" : {
"DeviceClass" : "Inverter",
"Scope" : "System"
},
"Status" : {
"Code" : 0,
"Reason" : "",
"UserMessage" : ""
},
"Timestamp" : "2020-07-28T10:01:31+02:00"
}
}
------------------------------------------------------------------------------
Acorn BBC Micro, model B with 6502 Second Processor.

DDRM
Posts: 184
Joined: Mon 02 Apr 2018, 18:04

Re: Parsing JSON files

Post by DDRM »

Just a note: I'm not sure why I'm getting moderation requests, which are delaying the appearance of some posts - I thought I had removed moderation from everybody. I will try to resolve this, but please don't feel "picked on"!

Best wishes,

David

User avatar
JeremyNicoll
Posts: 2
Joined: Sun 26 Jul 2020, 22:22
Location: Edinburgh

Re: Parsing JSON files

Post by JeremyNicoll »

I copied and pasted the sample data into the viewer at: http://jsonviewer.stack.hu/ as described before, which makes it far easier to see the structure.

It's not clear from what the OP posted whether the JSON string he sees contains line breaks. If it does, I'd remove them. Then you have a string like:

{ "Body" : { "Data" : { "DAY_ENERGY" : { "Unit" : "Wh", "Values" : { "1" : 4346 } }, "PAC" : { "Unit" : "W", "Values" : { "1" : 3105 } }, "TOTAL_ENERGY" : { "Unit" : "Wh", "Values" : { "1" : 6645459 } }, "YEAR_ENERGY" : { "Unit" : "Wh", "Values" : { "1" : 3938563 } } } }, "Head" : { "RequestArguments" : { "DeviceClass" : "Inverter", "Scope" : "System" }, "Status" : { "Code" : 0, "Reason" : "", "UserMessage" : "" }, "Timestamp" : "2020-07-28T10:01:31+02:00" } }

Replacing within that every instance of space colon space by colon, and every open-curly-bracket space by the bracket, and every space close-curly-bracket by that bracket gives:

{"Body":{"Data":{"DAY_ENERGY":{"Unit":"Wh", "Values":{"1":4346}}, "PAC":{"Unit":"W", "Values":{"1":3105}}, "TOTAL_ENERGY":{"Unit":"Wh", "Values":{"1":6645459}}, "YEAR_ENERGY":{"Unit":"Wh", "Values":{"1":3938563}}}}, "Head":{"RequestArguments":{"DeviceClass":"Inverter", "Scope":"System"}, "Status":{"Code":0, "Reason":"", "UserMessage":""}, "Timestamp":"2020-07-28T10:01:31+02:00"}}

/Of course/ if there are space-delimited colons or curly brackets in the meaningful data this is not altogether sensible. However for data like this it's probably pretty safe. That string, laid out a little differently is

{"Body":{"Data":{

followed by (split over four lines according to each named section)

"DAY_ENERGY":{"Unit":"Wh", "Values":{"1":4346}},
"PAC":{"Unit":"W", "Values":{"1":3105}},
"TOTAL_ENERGY":{"Unit":"Wh", "Values":{"1":6645459}},
"YEAR_ENERGY":{"Unit":"Wh", "Values":{"1":3938563}}}},

followed by

"Head":{"RequestArguments":{"DeviceClass":"Inverter", "Scope":"System"}, "Status":{"Code":0, "Reason":"", "UserMessage":""}, "Timestamp":"2020-07-28T10:01:31+02:00"}}

So if you transform the returned data as described, then find "DAY_ENERGY":{ and then find the position of the next }} and extract the string between those, you'd have

"Unit":"Wh", "Values":{"1":4346

Likewise, with "PAC":{ and }} you get "Unit":"W", "Values":{"1":3105

Likewise, with "TOTAL_ENERGY":{ and }} you get "Unit":"Wh", "Values":{"1":6645459

Likewise, with "YEAR_ENERGY":{ and }} you get "Unit":"Wh", "Values":{"1":3938563


If the format of the data returned is constant there's no need to write a 'proper' parser; it's easy to dig the meaningful numbers out of those substrings.

Mijndert Rooth
Posts: 5
Joined: Thu 23 Jul 2020, 14:38

Re: Parsing JSON files

Post by Mijndert Rooth »

JeremyNicoll wrote:
Fri 31 Jul 2020, 13:51
I copied and pasted the sample data into the viewer at: http://jsonviewer.stack.hu/ as described before, which makes it far easier to see the structure.

It's not clear from what the OP posted whether the JSON string he sees contains line breaks.
Jeremy,

Thanks for nicely breaking up the JSON output. This way it is easier to understand the structure and to find a way to extract the needed info.
And yes, the JSON output contains line-feeds (hex 0A). I have already made a procedure to leave out all the unnecessary stuff, so now only some procedures and/or functions to find what I need.

Thanks again for your effort,
Greetings.
Acorn BBC Micro, model B with 6502 Second Processor.