Topics

#rdf #issues #exiobase #issues #rdf #exiobase


Agneta
 

Hi everyone

Stefano and I were having a discussion on the current and prospective conversion of Exiobase to RDF. Last week we concluded that we will not include the location on flowobject in the exiobase data (look here for details) . 
With respect to this we chalked out the possible workflows.

Currently we use the exiobase SUT  obtained directly from the website, convert it to rdf, load it in the triplestore to be queried by mojo and converted to MRIO table using algoritms (for e.g. by-product technology model). The workflow is given in figure below:



What we are now suggesting is that the large exiobase MRIO tables could be broken down to individual country SUT and individual trade matrices (step 1). Each of these are to be converted into rdf triples (step 2). imported to the triplestore (step 3). Queried by MOJO (Step 4). Apply the necessary algorithms (Step 5) and import it back to the triplestore (Step 6). Please see the diagram below and check if that is correct.  

Specific questions:
  1. Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano. 
  2. Also at what point do we want arrange the data in tidy format instead of a matrix? 


 

If the workflow seems ok Stefano is ready to work on the MOJO repository.


Bo Weidema
 

Den 2019-10-30 kl. 16.07 skrev Agneta:

Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano.
Why do the EXIOBASE consortium not just publish these on the website, saving us the hazzle of re-constructing the original data?

Bo


Konstantin Stadler
 

Hi Bo/2.0 LCA Team

You (trough Stefano) have access to the repository having the country SUT tables. I can add more people to the box account if needed.

Best
Konstantin

On 30/10/2019 17:35, Bo Weidema wrote:
Den 2019-10-30 kl. 16.07 skrev Agneta:

Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano.
Why do the EXIOBASE consortium not just publish these on the website, saving us the hazzle of re-constructing the original data?
Bo


Bo Weidema
 

I am aware that we have priority access as consortium members. I was hinting to making this publicly available...

Bo

Den 2019-11-01 kl. 15.28 skrev Konstantin Stadler:

Hi Bo/2.0 LCA Team

You (trough Stefano) have access to the repository having the country SUT tables. I can add more people to the box account if needed.

Best
Konstantin

On 30/10/2019 17:35, Bo Weidema wrote:
Den 2019-10-30 kl. 16.07 skrev Agneta:

Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano.
Why do the EXIOBASE consortium not just publish these on the website, saving us the hazzle of re-constructing the original data?

Bo






--


Konstantin Stadler
 

These folders are as public as the updates of EXIOBASE (for everyone who contacts us) - they are in the public folder. I know that this is not the most elegant solution (not much I can do...).

On 01/11/2019 15:45, Bo Weidema wrote:
I am aware that we have priority access as consortium members. I was hinting to making this publicly available...
Bo
Den 2019-11-01 kl. 15.28 skrev Konstantin Stadler:
Hi Bo/2.0 LCA Team

You (trough Stefano) have access to the repository having the country SUT tables. I can add more people to the box account if needed.

Best
Konstantin

On 30/10/2019 17:35, Bo Weidema wrote:
Den 2019-10-30 kl. 16.07 skrev Agneta:

Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano.
Why do the EXIOBASE consortium not just publish these on the website, saving us the hazzle of re-constructing the original data?

Bo



--


Bo Weidema
 

Dear Konstantin,

This is good news to me! This implies that anyone who cares can re-publish them under the same license...

Best regards

Bo

Den 2019-11-01 kl. 15.49 skrev Konstantin Stadler:

These folders are as public as the updates of EXIOBASE (for everyone who contacts us) - they are in the public folder. I know that this is not the most elegant solution (not much I can do...).


On 01/11/2019 15:45, Bo Weidema wrote:
I am aware that we have priority access as consortium members. I was hinting to making this publicly available...

Bo

Den 2019-11-01 kl. 15.28 skrev Konstantin Stadler:
Hi Bo/2.0 LCA Team

You (trough Stefano) have access to the repository having the country SUT tables. I can add more people to the box account if needed.

Best
Konstantin

On 30/10/2019 17:35, Bo Weidema wrote:
Den 2019-10-30 kl. 16.07 skrev Agneta:

Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano.
Why do the EXIOBASE consortium not just publish these on the website, saving us the hazzle of re-constructing the original data?

Bo






-- 




--


Konstantin Stadler
 

Hi Bo,

That's muddy waters. The version on exiobase.eu has a licence which applies to the data there. This is the outcome of the EU-funded project.

The data on box is without licence. We worked on that without funding and share it with colleagues bona fides - either for collaborations on research or b/c they contributed to the db. The bona fides part is that they apply the same rules before sharing the data further.

Anyway, the details here are something not in my hand - at the end the consortium (afak including 2.0 LCA) needs to decide. Perhaps have a chat with Richard/Arnold?

Best
kst



On 1 November 2019 19:09:39 CET, Bo Weidema <bo.weidema@...> wrote:

Dear Konstantin,

This is good news to me! This implies that anyone who cares can re-publish them under the same license...

Best regards

Bo

Den 2019-11-01 kl. 15.49 skrev Konstantin Stadler:
These folders are as public as the updates of EXIOBASE (for everyone who contacts us) - they are in the public folder. I know that this is not the most elegant solution (not much I can do...).


On 01/11/2019 15:45, Bo Weidema wrote:
I am aware that we have priority access as consortium members. I was hinting to making this publicly available...

Bo

Den 2019-11-01 kl. 15.28 skrev Konstantin Stadler:
Hi Bo/2.0 LCA Team

You (trough Stefano) have access to the repository having the country SUT tables. I can add more people to the box account if needed.

Best
Konstantin

On 30/10/2019 17:35, Bo Weidema wrote:
Den 2019-10-30 kl. 16.07 skrev Agneta:

Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano.
Why do the EXIOBASE consortium not just publish these on the website, saving us the hazzle of re-constructing the original data?

Bo






-- 




--


Stefano Merciai
 

Hi all,

I can provide the Exiobase hybrid version of national tables but I wonder whether this is the way to go. I try to explain it better.

Exiobase is a multi-regional IO database, therefore on the website you can only download MR-tables. Of course who contributes to Exiobase has as intermediate result the national tables stored somewhere but not on the official Exiobase website. But then, does Bonsai rely on publicly available data or on 'confidential' data? In the light of future updates, what is the best way to go?

And what if we intend to use other world IO databases such as the WIOD? We will have the same problem.

However, moving from MR-tables to a national table is a quite simple procedure because we just need to aggregate rows and columns. This procedure could be easily generalized for future imports of multi-regional databases. The only issue I see here is that Exiobase will be stored twice on the Bonsai storage.

Best,

Stefano




On 01/11/2019 15:45, Bo Weidema wrote:

I am aware that we have priority access as consortium members. I was hinting to making this publicly available...

Bo

Den 2019-11-01 kl. 15.28 skrev Konstantin Stadler:
Hi Bo/2.0 LCA Team

You (trough Stefano) have access to the repository having the country SUT tables. I can add more people to the box account if needed.

Best
Konstantin

On 30/10/2019 17:35, Bo Weidema wrote:
Den 2019-10-30 kl. 16.07 skrev Agneta:

Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano.
Why do the EXIOBASE consortium not just publish these on the website, saving us the hazzle of re-constructing the original data?

Bo






--

--


Bo Weidema
 

Dear Stefano,

Thanks for your reflections. In the long run, BONSAI should of course not rely on Exiobase or any other processed data, but on as a "raw" data source as possible. For IO tables, this means national Supply Use data, obtained directly from the national statistical agencies etc. Back-calulating these from Exiobase would in all cases just be a temporary solution to get a good start with a relatively complete database.

Best regards

Bo

Den 2019-11-04 kl. 11.16 skrev Stefano Merciai:

Hi all,

I can provide the Exiobase hybrid version of national tables but I wonder whether this is the way to go. I try to explain it better.

Exiobase is a multi-regional IO database, therefore on the website you can only download MR-tables. Of course who contributes to Exiobase has as intermediate result the national tables stored somewhere but not on the official Exiobase website. But then, does Bonsai rely on publicly available data or on 'confidential' data? In the light of future updates, what is the best way to go?

And what if we intend to use other world IO databases such as the WIOD? We will have the same problem.

However, moving from MR-tables to a national table is a quite simple procedure because we just need to aggregate rows and columns. This procedure could be easily generalized for future imports of multi-regional databases. The only issue I see here is that Exiobase will be stored twice on the Bonsai storage.

Best,

Stefano




On 01/11/2019 15:45, Bo Weidema wrote:

I am aware that we have priority access as consortium members. I was hinting to making this publicly available...

Bo

Den 2019-11-01 kl. 15.28 skrev Konstantin Stadler:
Hi Bo/2.0 LCA Team

You (trough Stefano) have access to the repository having the country SUT tables. I can add more people to the box account if needed.

Best
Konstantin

On 30/10/2019 17:35, Bo Weidema wrote:
Den 2019-10-30 kl. 16.07 skrev Agneta:

Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano.
Why do the EXIOBASE consortium not just publish these on the website, saving us the hazzle of re-constructing the original data?

Bo






--

--
--


 

Sorry for chiming in late here.

+1 to the idea of only using the publicly available data, it seems to
me a very unnecessary and dangerous step to start using private (and
unlicensed!) data at the beginning of our journey.

I have, after some effort, convert the hybrid IO table to a tidy
format, with proper packaging and metadata in the data package format.
The data is here:
http://files.brightwaylca.org/exiobase-3.3.17-hybrid.tar, the code is
here: https://github.com/brightway-lca/mrio_common_metadata/tree/master/mrio_common_metadata/conversion/exiobase_3_hybrid.
I would be happy to do this for the SU tables as well.

Using the tidy format, and standards packaging, would allow us to have
one importer that would work for multiple MRIO/SU databases, which
seems to me to be a big win.

Agneta, your diagrams are great, but could you explain *why* we would
want to separate out the individual country tables? Is this just to
get the raw country data before it is processed by the EXIOBASE
balancing algorithm? Separating out the country tables from the
provided data seems easy but unnecessary.

-C

On Mon, 4 Nov 2019 at 16:35, Bo Weidema <bo.weidema@...> wrote:

Dear Stefano,

Thanks for your reflections. In the long run, BONSAI should of course not rely on Exiobase or any other processed data, but on as a "raw" data source as possible. For IO tables, this means national Supply Use data, obtained directly from the national statistical agencies etc. Back-calulating these from Exiobase would in all cases just be a temporary solution to get a good start with a relatively complete database.

Best regards

Bo

Den 2019-11-04 kl. 11.16 skrev Stefano Merciai:

Hi all,

I can provide the Exiobase hybrid version of national tables but I wonder whether this is the way to go. I try to explain it better.

Exiobase is a multi-regional IO database, therefore on the website you can only download MR-tables. Of course who contributes to Exiobase has as intermediate result the national tables stored somewhere but not on the official Exiobase website. But then, does Bonsai rely on publicly available data or on 'confidential' data? In the light of future updates, what is the best way to go?

And what if we intend to use other world IO databases such as the WIOD? We will have the same problem.

However, moving from MR-tables to a national table is a quite simple procedure because we just need to aggregate rows and columns. This procedure could be easily generalized for future imports of multi-regional databases. The only issue I see here is that Exiobase will be stored twice on the Bonsai storage.

Best,

Stefano




On 01/11/2019 15:45, Bo Weidema wrote:

I am aware that we have priority access as consortium members. I was hinting to making this publicly available...

Bo

Den 2019-11-01 kl. 15.28 skrev Konstantin Stadler:

Hi Bo/2.0 LCA Team

You (trough Stefano) have access to the repository having the country SUT tables. I can add more people to the box account if needed.

Best
Konstantin

On 30/10/2019 17:35, Bo Weidema wrote:

Den 2019-10-30 kl. 16.07 skrev Agneta:

Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano.

Why do the EXIOBASE consortium not just publish these on the website, saving us the hazzle of re-constructing the original data?

Bo






--


--

--
--
############################
Chris Mutel
Technology Assessment Group, LEA
Paul Scherrer Institut
OHSA D22
5232 Villigen PSI
Switzerland
http://chris.mutel.org
Telefon: +41 56 310 5787
############################


Matteo Lissandrini (AAU)
 

Hi Chris,


unfortunately half of what you say is alien to me, but I have the impression that this is about the numeric data that the RDF converter is handling.


Will this require to write from scratch the Exiobase RDF converter? do I understand correctly or is this about some other data?


Thanks,

Matteo


---
Matteo Lissandrini

Department of Computer Science
Aalborg University

http://people.cs.aau.dk/~matteo






From: main@bonsai.groups.io <main@bonsai.groups.io> on behalf of Chris Mutel via Groups.Io <cmutel@...>
Sent: Wednesday, November 6, 2019 10:33:31 AM
To: main@bonsai.groups.io
Subject: Re: [bonsai] #rdf #issues #exiobase
 
Sorry for chiming in late here.

+1 to the idea of only using the publicly available data, it seems to
me a very unnecessary and dangerous step to start using private (and
unlicensed!) data at the beginning of our journey.

I have, after some effort, convert the hybrid IO table to a tidy
format, with proper packaging and metadata in the data package format.
The data is here:
http://files.brightwaylca.org/exiobase-3.3.17-hybrid.tar, the code is
here: https://github.com/brightway-lca/mrio_common_metadata/tree/master/mrio_common_metadata/conversion/exiobase_3_hybrid.
I would be happy to do this for the SU tables as well.

Using the tidy format, and standards packaging, would allow us to have
one importer that would work for multiple MRIO/SU databases, which
seems to me to be a big win.

Agneta, your diagrams are great, but could you explain *why* we would
want to separate out the individual country tables? Is this just to
get the raw country data before it is processed by the EXIOBASE
balancing algorithm? Separating out the country tables from the
provided data seems easy but unnecessary.

-C

On Mon, 4 Nov 2019 at 16:35, Bo Weidema <bo.weidema@...> wrote:
>
> Dear Stefano,
>
> Thanks for your reflections. In the long run, BONSAI should of course not rely on Exiobase or any other processed data, but on as a "raw" data source as possible. For IO tables, this means national Supply Use data, obtained directly from the national statistical agencies etc. Back-calulating these from Exiobase would in all cases just be a temporary solution to get a good start with a relatively complete database.
>
> Best regards
>
> Bo
>
> Den 2019-11-04 kl. 11.16 skrev Stefano Merciai:
>
> Hi all,
>
> I can provide the Exiobase hybrid version of national tables but I wonder whether this is the way to go. I try to explain it better.
>
> Exiobase is a multi-regional IO database, therefore on the website you can only download MR-tables. Of course who contributes to Exiobase has as intermediate result the national tables stored somewhere but not on the official Exiobase website. But then, does Bonsai rely on publicly available data or on 'confidential' data? In the light of future updates, what is the best way to go?
>
> And what if we intend to use other world IO databases such as the WIOD? We will have the same problem.
>
> However, moving from MR-tables to a national table is a quite simple procedure because we just need to aggregate rows and columns. This procedure could be easily generalized for future imports of multi-regional databases. The only issue I see here is that Exiobase will be stored twice on the Bonsai storage.
>
> Best,
>
> Stefano
>
>
>
>
> On 01/11/2019 15:45, Bo Weidema wrote:
>
> I am aware that we have priority access as consortium members. I was hinting to making this publicly available...
>
> Bo
>
> Den 2019-11-01 kl. 15.28 skrev Konstantin Stadler:
>
> Hi Bo/2.0 LCA Team
>
> You (trough Stefano) have access to the repository having the country SUT tables. I can add more people to the box account if needed.
>
> Best
> Konstantin
>
> On 30/10/2019 17:35, Bo Weidema wrote:
>
> Den 2019-10-30 kl. 16.07 skrev Agneta:
>
> Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano.
>
> Why do the EXIOBASE consortium not just publish these on the website, saving us the hazzle of re-constructing the original data?
>
> Bo
>
>
>
>
>
>
> --
>
>
> --
>
> --
>



--
############################
Chris Mutel
Technology Assessment Group, LEA
Paul Scherrer Institut
OHSA D22
5232 Villigen PSI
Switzerland
http://chris.mutel.org
Telefon: +41 56 310 5787
############################




Agneta
 

Hi Chris

The disaggregation to country level data is mainly to avoid the assumptions made using the trade matrix (info on markets for different flow objects). Current SUT tables include this information, given as location of flow objects. This information is not necessary for the raw SUT data. Assumptions with respect to trade matrix can be added later as I discussed with Stefano. 
 The discussion on this was elaborated by Bo and updated on the read me for the MOJO repository ( https://github.com/BONSAMURAIS/mojo#why-flows-do-not-have-a-location

Best 
Agneta 

On Wed, 6 Nov 2019, 10:34 AM Chris Mutel, <cmutel@...> wrote:
Sorry for chiming in late here.

+1 to the idea of only using the publicly available data, it seems to
me a very unnecessary and dangerous step to start using private (and
unlicensed!) data at the beginning of our journey.

I have, after some effort, convert the hybrid IO table to a tidy
format, with proper packaging and metadata in the data package format.
The data is here:
http://files.brightwaylca.org/exiobase-3.3.17-hybrid.tar, the code is
here: https://github.com/brightway-lca/mrio_common_metadata/tree/master/mrio_common_metadata/conversion/exiobase_3_hybrid.
I would be happy to do this for the SU tables as well.

Using the tidy format, and standards packaging, would allow us to have
one importer that would work for multiple MRIO/SU databases, which
seems to me to be a big win.

Agneta, your diagrams are great, but could you explain *why* we would
want to separate out the individual country tables? Is this just to
get the raw country data before it is processed by the EXIOBASE
balancing algorithm? Separating out the country tables from the
provided data seems easy but unnecessary.

-C

On Mon, 4 Nov 2019 at 16:35, Bo Weidema <bo.weidema@...> wrote:
>
> Dear Stefano,
>
> Thanks for your reflections. In the long run, BONSAI should of course not rely on Exiobase or any other processed data, but on as a "raw" data source as possible. For IO tables, this means national Supply Use data, obtained directly from the national statistical agencies etc. Back-calulating these from Exiobase would in all cases just be a temporary solution to get a good start with a relatively complete database.
>
> Best regards
>
> Bo
>
> Den 2019-11-04 kl. 11.16 skrev Stefano Merciai:
>
> Hi all,
>
> I can provide the Exiobase hybrid version of national tables but I wonder whether this is the way to go. I try to explain it better.
>
> Exiobase is a multi-regional IO database, therefore on the website you can only download MR-tables. Of course who contributes to Exiobase has as intermediate result the national tables stored somewhere but not on the official Exiobase website. But then, does Bonsai rely on publicly available data or on 'confidential' data? In the light of future updates, what is the best way to go?
>
> And what if we intend to use other world IO databases such as the WIOD? We will have the same problem.
>
> However, moving from MR-tables to a national table is a quite simple procedure because we just need to aggregate rows and columns. This procedure could be easily generalized for future imports of multi-regional databases. The only issue I see here is that Exiobase will be stored twice on the Bonsai storage.
>
> Best,
>
> Stefano
>
>
>
>
> On 01/11/2019 15:45, Bo Weidema wrote:
>
> I am aware that we have priority access as consortium members. I was hinting to making this publicly available...
>
> Bo
>
> Den 2019-11-01 kl. 15.28 skrev Konstantin Stadler:
>
> Hi Bo/2.0 LCA Team
>
> You (trough Stefano) have access to the repository having the country SUT tables. I can add more people to the box account if needed.
>
> Best
> Konstantin
>
> On 30/10/2019 17:35, Bo Weidema wrote:
>
> Den 2019-10-30 kl. 16.07 skrev Agneta:
>
> Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano.
>
> Why do the EXIOBASE consortium not just publish these on the website, saving us the hazzle of re-constructing the original data?
>
> Bo
>
>
>
>
>
>
> --
>
>
> --
>
> --
>



--
############################
Chris Mutel
Technology Assessment Group, LEA
Paul Scherrer Institut
OHSA D22
5232 Villigen PSI
Switzerland
http://chris.mutel.org
Telefon: +41 56 310 5787
############################




 

On Wed, 6 Nov 2019 at 11:42, Agneta <@Agneta.Ghose> wrote:
The disaggregation to country level data is mainly to avoid the assumptions made using the trade matrix (info on markets for different flow objects). Current SUT tables include this information, given as location of flow objects.
OK, but we can just... ignore... this location? Each national S or U
table by definition only has one location anyway.

In any case, I don't think there needs to be a discussion here, as our
ontology does not allow flow objects to have a location, so we can't
input this data in the first place.

This information is not necessary for the raw SUT data. Assumptions with respect to trade matrix can be added later as I discussed with Stefano.
The discussion on this was elaborated by Bo and updated on the read me for the MOJO repository ( https://github.com/BONSAMURAIS/mojo#why-flows-do-not-have-a-location)

Best
Agneta

On Wed, 6 Nov 2019, 10:34 AM Chris Mutel, <@cmutel> wrote:

Sorry for chiming in late here.

+1 to the idea of only using the publicly available data, it seems to
me a very unnecessary and dangerous step to start using private (and
unlicensed!) data at the beginning of our journey.

I have, after some effort, convert the hybrid IO table to a tidy
format, with proper packaging and metadata in the data package format.
The data is here:
http://files.brightwaylca.org/exiobase-3.3.17-hybrid.tar, the code is
here: https://github.com/brightway-lca/mrio_common_metadata/tree/master/mrio_common_metadata/conversion/exiobase_3_hybrid.
I would be happy to do this for the SU tables as well.

Using the tidy format, and standards packaging, would allow us to have
one importer that would work for multiple MRIO/SU databases, which
seems to me to be a big win.

Agneta, your diagrams are great, but could you explain *why* we would
want to separate out the individual country tables? Is this just to
get the raw country data before it is processed by the EXIOBASE
balancing algorithm? Separating out the country tables from the
provided data seems easy but unnecessary.

-C

On Mon, 4 Nov 2019 at 16:35, Bo Weidema <bo.weidema@...> wrote:

Dear Stefano,

Thanks for your reflections. In the long run, BONSAI should of course not rely on Exiobase or any other processed data, but on as a "raw" data source as possible. For IO tables, this means national Supply Use data, obtained directly from the national statistical agencies etc. Back-calulating these from Exiobase would in all cases just be a temporary solution to get a good start with a relatively complete database.

Best regards

Bo

Den 2019-11-04 kl. 11.16 skrev Stefano Merciai:

Hi all,

I can provide the Exiobase hybrid version of national tables but I wonder whether this is the way to go. I try to explain it better.

Exiobase is a multi-regional IO database, therefore on the website you can only download MR-tables. Of course who contributes to Exiobase has as intermediate result the national tables stored somewhere but not on the official Exiobase website. But then, does Bonsai rely on publicly available data or on 'confidential' data? In the light of future updates, what is the best way to go?

And what if we intend to use other world IO databases such as the WIOD? We will have the same problem.

However, moving from MR-tables to a national table is a quite simple procedure because we just need to aggregate rows and columns. This procedure could be easily generalized for future imports of multi-regional databases. The only issue I see here is that Exiobase will be stored twice on the Bonsai storage.

Best,

Stefano




On 01/11/2019 15:45, Bo Weidema wrote:

I am aware that we have priority access as consortium members. I was hinting to making this publicly available...

Bo

Den 2019-11-01 kl. 15.28 skrev Konstantin Stadler:

Hi Bo/2.0 LCA Team

You (trough Stefano) have access to the repository having the country SUT tables. I can add more people to the box account if needed.

Best
Konstantin

On 30/10/2019 17:35, Bo Weidema wrote:

Den 2019-10-30 kl. 16.07 skrev Agneta:

Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano.

Why do the EXIOBASE consortium not just publish these on the website, saving us the hazzle of re-constructing the original data?

Bo






--


--

--


--
############################
Chris Mutel
Technology Assessment Group, LEA
Paul Scherrer Institut
OHSA D22
5232 Villigen PSI
Switzerland
http://chris.mutel.org
Telefon: +41 56 310 5787
############################




--
############################
Chris Mutel
Technology Assessment Group, LEA
Paul Scherrer Institut
OHSA D22
5232 Villigen PSI
Switzerland
http://chris.mutel.org
Telefon: +41 56 310 5787
############################


 

On Wed, 6 Nov 2019 at 11:30, Matteo Lissandrini (AAU) <matteo@...> wrote:
unfortunately half of what you say is alien to me, but I have the impression that this is about the numeric data that the RDF converter is handling.
Indeed, this was not clear. Sorry. I was referring to
https://frictionlessdata.io/specs/data-package/ and
https://frictionlessdata.io/specs/tabular-data-resource/ and
https://en.wikipedia.org/wiki/Tidy_data; see also attached the
datapackage.json file for the IO tables.

Will this require to write from scratch the Exiobase RDF converter? do I understand correctly or is this about some other data?
No, the "intermediate" file is basically the same;
https://github.com/BONSAMURAIS/EXIOBASE-conversion-software/blob/master/scripts/excel2csv.py
does more or less the same thing, just less completely (not all
extensions, doesn't check to make sure EXIOBASE authors were
consistent in row and column labels, doesn't correct typos or clean
whitespace), and without providing metadata following a specification.
But we need to update this software anyway to a) make it a proper
installable package, b) follow the URI schema that we are now using,
and c) fill in all the "TODO"s in the code.


Matteo Lissandrini (AAU)
 

Hi Chris,


for how the data is now in the publicly available excel file, we can ignore for the SUPPLY, but we cannot "just ignore" the location for the USE table, we need to recompute the aggregates:


See https://github.com/BONSAMURAIS/rdf/issues/4


---
Matteo Lissandrini

Department of Computer Science
Aalborg University

http://people.cs.aau.dk/~matteo






From: main@bonsai.groups.io <main@bonsai.groups.io> on behalf of Chris Mutel via Groups.Io <cmutel@...>
Sent: Wednesday, November 6, 2019 12:01:39 PM
To: main@bonsai.groups.io
Subject: Re: [bonsai] #rdf #issues #exiobase
 
On Wed, 6 Nov 2019 at 11:42, Agneta <agneta.20@...> wrote:
> The disaggregation to country level data is mainly to avoid the assumptions made using the trade matrix (info on markets for different flow objects). Current SUT tables include this information, given as location of flow objects.

OK, but we can just... ignore... this location? Each national S or U
table by definition only has one location anyway.

In any case, I don't think there needs to be a discussion here, as our
ontology does not allow flow objects to have a location, so we can't
input this data in the first place.

> This information is not necessary for the raw SUT data. Assumptions with respect to trade matrix can be added later as I discussed with Stefano.
>  The discussion on this was elaborated by Bo and updated on the read me for the MOJO repository ( https://github.com/BONSAMURAIS/mojo#why-flows-do-not-have-a-location)
>
> Best
> Agneta
>
> On Wed, 6 Nov 2019, 10:34 AM Chris Mutel, <cmutel@...> wrote:
>>
>> Sorry for chiming in late here.
>>
>> +1 to the idea of only using the publicly available data, it seems to
>> me a very unnecessary and dangerous step to start using private (and
>> unlicensed!) data at the beginning of our journey.
>>
>> I have, after some effort, convert the hybrid IO table to a tidy
>> format, with proper packaging and metadata in the data package format.
>> The data is here:
>> http://files.brightwaylca.org/exiobase-3.3.17-hybrid.tar, the code is
>> here: https://github.com/brightway-lca/mrio_common_metadata/tree/master/mrio_common_metadata/conversion/exiobase_3_hybrid.
>> I would be happy to do this for the SU tables as well.
>>
>> Using the tidy format, and standards packaging, would allow us to have
>> one importer that would work for multiple MRIO/SU databases, which
>> seems to me to be a big win.
>>
>> Agneta, your diagrams are great, but could you explain *why* we would
>> want to separate out the individual country tables? Is this just to
>> get the raw country data before it is processed by the EXIOBASE
>> balancing algorithm? Separating out the country tables from the
>> provided data seems easy but unnecessary.
>>
>> -C
>>
>> On Mon, 4 Nov 2019 at 16:35, Bo Weidema <bo.weidema@...> wrote:
>> >
>> > Dear Stefano,
>> >
>> > Thanks for your reflections. In the long run, BONSAI should of course not rely on Exiobase or any other processed data, but on as a "raw" data source as possible. For IO tables, this means national Supply Use data, obtained directly from the national statistical agencies etc. Back-calulating these from Exiobase would in all cases just be a temporary solution to get a good start with a relatively complete database.
>> >
>> > Best regards
>> >
>> > Bo
>> >
>> > Den 2019-11-04 kl. 11.16 skrev Stefano Merciai:
>> >
>> > Hi all,
>> >
>> > I can provide the Exiobase hybrid version of national tables but I wonder whether this is the way to go. I try to explain it better.
>> >
>> > Exiobase is a multi-regional IO database, therefore on the website you can only download MR-tables. Of course who contributes to Exiobase has as intermediate result the national tables stored somewhere but not on the official Exiobase website. But then, does Bonsai rely on publicly available data or on 'confidential' data? In the light of future updates, what is the best way to go?
>> >
>> > And what if we intend to use other world IO databases such as the WIOD? We will have the same problem.
>> >
>> > However, moving from MR-tables to a national table is a quite simple procedure because we just need to aggregate rows and columns. This procedure could be easily generalized for future imports of multi-regional databases. The only issue I see here is that Exiobase will be stored twice on the Bonsai storage.
>> >
>> > Best,
>> >
>> > Stefano
>> >
>> >
>> >
>> >
>> > On 01/11/2019 15:45, Bo Weidema wrote:
>> >
>> > I am aware that we have priority access as consortium members. I was hinting to making this publicly available...
>> >
>> > Bo
>> >
>> > Den 2019-11-01 kl. 15.28 skrev Konstantin Stadler:
>> >
>> > Hi Bo/2.0 LCA Team
>> >
>> > You (trough Stefano) have access to the repository having the country SUT tables. I can add more people to the box account if needed.
>> >
>> > Best
>> > Konstantin
>> >
>> > On 30/10/2019 17:35, Bo Weidema wrote:
>> >
>> > Den 2019-10-30 kl. 16.07 skrev Agneta:
>> >
>> > Do we need to have a separate repo that breaks the exiobase SUT from the website ? We could also get these country specific tables directly from Stefano.
>> >
>> > Why do the EXIOBASE consortium not just publish these on the website, saving us the hazzle of re-constructing the original data?
>> >
>> > Bo
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> >
>> >
>> > --
>> >
>> > --
>> >
>>
>>
>>
>> --
>> ############################
>> Chris Mutel
>> Technology Assessment Group, LEA
>> Paul Scherrer Institut
>> OHSA D22
>> 5232 Villigen PSI
>> Switzerland
>> http://chris.mutel.org
>> Telefon: +41 56 310 5787
>> ############################
>>
>>
>>
>



--
############################
Chris Mutel
Technology Assessment Group, LEA
Paul Scherrer Institut
OHSA D22
5232 Villigen PSI
Switzerland
http://chris.mutel.org
Telefon: +41 56 310 5787
############################




Matteo Lissandrini (AAU)
 


>> Will this require to write from scratch the Exiobase RDF converter? do I understand correctly or is this about some other data?

> But we need to update this software anyway to a) make it a proper
> installable package, b) follow the URI schema that we are now using,
> and c) fill in all the "TODO"s in the code.

Ok, I had the task to update this script for the USE table, as per other email, but then I'm blocked until I have the new output.

This means that for now, the published data is only for the supply table.

We will need to re-sync for completing this work.

Thanks,
Matteo


 

Sorry, I don't want to be a pain in the ass, but I think we are going
a bit down the raod of Stalin's ideological purity here... In an ideal
world, we would have separate sources of trade data, and could ignore
the EXIOBASE trade "assumptions"; but in this ideal world, we could
also take the SU tables directly from each country. EXIOBASE has the
trade information, and it is balanced. We need this trade data, and
don't really want to start a whole new project to get it from another
source (and clean it!). Bo says that "In a true SUT, the flows enter
and leave an activity but do not yet have information on their origin
and destination," but EXIOBASE is not just a SUT, it is also trade
data.

"The EXIOBASE SUT is overspecified in this sense that it already has
interpreted the information in the trade statistics in a specific
(attributional) way. This error should not be imported into the BONSAI
implementation, which should leave the user free to link SUT
activities with different linking algorithms." But we are free to
(re-)link SUT activities with different linking algorithms, even if we
import this data! All data is BONSAI are factual claims that we can
use or ignore as we wish.

We go here to a fundamental decision for the entire project, namely:
Should we let our collective or individual biases lead to data
modification **before** it enters the system? It was my impression
that our consensus decision from the hackathon was that we do not
alter or delete data before it enters the system, unless such
modification would never be controversial in any way (i.e. unit
conversions or changing labels in cases where there is zero
ambiguity). Did this change? I don't accept that it changed in a
comment in a Github issue where two people reported that they
discussed something offline.

On Wed, 6 Nov 2019 at 13:27, Matteo Lissandrini (AAU) <matteo@...> wrote:


Will this require to write from scratch the Exiobase RDF converter? do I understand correctly or is this about some other data?
But we need to update this software anyway to a) make it a proper
installable package, b) follow the URI schema that we are now using,
and c) fill in all the "TODO"s in the code.
Ok, I had the task to update this script for the USE table, as per other email, but then I'm blocked until I have the new output.

This means that for now, the published data is only for the supply table.

We will need to re-sync for completing this work.

Thanks,
Matteo
--
############################
Chris Mutel
Technology Assessment Group, LEA
Paul Scherrer Institut
OHSA D22
5232 Villigen PSI
Switzerland
http://chris.mutel.org
Telefon: +41 56 310 5787
############################


Bo Weidema
 

Dear Chris,

The issue is not about interpreting or deleting data, but simply to make the import of it consistent with our ontology. The Exiobase SUT data includes two kinds of data 1) inputs and outputs of industries ("production activities") 2) inputs and outputs of bi-lateral trade ("market activities"). Both kinds of flow data being SUT data, they do not have location as a property, i.e. the input flows are not linked to an origin, and the output flows are not linked to a destination. This linking is what happens when we produce the Direct Requirement Matrix using the product system algorithms on the SUT.

Since the Exiobase SUT has a convention of integrating the bi-lateral market data in the data for each industry, our import algorithm needs to separate these two kinds of data, to make them consistent with the ontology, but more importantly to make them useful for the later linking.

This is done by:

1) Placing the information on bi-lateral trade flows in their respective market activities (for each of the 169 products, there are 49*49 bi-lateral markets, many of which will be empty (having no flows), and may therefore be ignored). This takes care of the disaggregated import data of the industries.

2) Aggregating the disaggregated import data of the industries, so that each industry only have 169 imported products, not the current 49*169 (since that information is already present in the above bi-lateral trade data).

This way of organising the Exiobase import preserves all data intact, and now in a more meaningful format that allows use by any relevant product system algorithm.

Of course, this transformation is completely transparent and an alternative could be to make an Exiobase ontology term for this "exiobase:import origin" and use this for importing the Exiobase data to RDF with this term attached and then do the "stripping" to BONSAI ontology in RDF. However, this would create precedence for making RDF ontologies for all other strange data formats that poeople wish to provide and making RDF converters for these. I do not think that is a road that we would want to go down. The whole purpose of the BONSAI ontology is to be lean and nevertheless complete enough to allow loss-less import of all the different kind of data people may wish to provide.

I hope this clarifies the reason for staying with the BONSAI ontology on this point and to adapt the import so that loss-less imprt of Exiobase nevertheless is possible.

Best regards

Bo

Den 2019-11-07 kl. 12.28 skrev Chris Mutel:

Sorry, I don't want to be a pain in the ass, but I think we are going
a bit down the raod of Stalin's ideological purity here... In an ideal
world, we would have separate sources of trade data, and could ignore
the EXIOBASE trade "assumptions"; but in this ideal world, we could
also take the SU tables directly from each country. EXIOBASE has the
trade information, and it is balanced. We need this trade data, and
don't really want to start a whole new project to get it from another
source (and clean it!). Bo says that "In a true SUT, the flows enter
and leave an activity but do not yet have information on their origin
and destination," but EXIOBASE is not just a SUT, it is also trade
data.

"The EXIOBASE SUT is overspecified in this sense that it already has
interpreted the information in the trade statistics in a specific
(attributional) way. This error should not be imported into the BONSAI
implementation, which should leave the user free to link SUT
activities with different linking algorithms." But we are free to
(re-)link SUT activities with different linking algorithms, even if we
import this data! All data is BONSAI are factual claims that we can
use or ignore as we wish.

We go here to a fundamental decision for the entire project, namely:
Should we let our collective or individual biases lead to data
modification **before** it enters the system? It was my impression
that our consensus decision from the hackathon was that we do not
alter or delete data before it enters the system, unless such
modification would never be controversial in any way (i.e. unit
conversions or changing labels in cases where there is zero
ambiguity). Did this change? I don't accept that it changed in a
comment in a Github issue where two people reported that they
discussed something offline.

On Wed, 6 Nov 2019 at 13:27, Matteo Lissandrini (AAU) <matteo@...> wrote:

Will this require to write from scratch the Exiobase RDF converter? do I understand correctly or is this about some other data?

        
But we need to update this software anyway to a) make it a proper
installable package, b) follow the URI schema that we are now using,
and c) fill in all the "TODO"s in the code.
Ok, I had the task to update this script for the USE table, as per other email, but then I'm blocked until I have the new output.

This means that for now, the published data is only for the supply table.

We will need to re-sync for completing this work.

Thanks,
Matteo



--


Matteo Lissandrini (AAU)
 

Hi all,


I am not in the position of taking sides.

I just want to provide one clarification here.


Both options (keep the data as is in Exiobase, or re-aggregate) are compatible with the ontology. I would say we should be quite happy with this result per se, is not a minor feat! :)


Both option require an extra step when importing the data:

 - keep the data as is requires the instantiation of the "activity" that links the flow with the location so that the flow is an output of that activity, I say instantiation, this is the same thing we do when instantiating the URI for Paddy-rice or for Rest of Europe. So this is not a change in the ontology

 - re-aggregate requires during conversion to take all the splitted flows and to sum  them up in a single flow.



Cheers,

Matteo







---
Matteo Lissandrini

Department of Computer Science
Aalborg University

http://people.cs.aau.dk/~matteo






From: main@bonsai.groups.io <main@bonsai.groups.io> on behalf of Bo Weidema via Groups.Io <bo.weidema@...>
Sent: Thursday, November 7, 2019 3:01:32 PM
To: main@bonsai.groups.io
Subject: Re: [bonsai] #rdf #issues #exiobase
 

Dear Chris,

The issue is not about interpreting or deleting data, but simply to make the import of it consistent with our ontology. The Exiobase SUT data includes two kinds of data 1) inputs and outputs of industries ("production activities") 2) inputs and outputs of bi-lateral trade ("market activities"). Both kinds of flow data being SUT data, they do not have location as a property, i.e. the input flows are not linked to an origin, and the output flows are not linked to a destination. This linking is what happens when we produce the Direct Requirement Matrix using the product system algorithms on the SUT.

Since the Exiobase SUT has a convention of integrating the bi-lateral market data in the data for each industry, our import algorithm needs to separate these two kinds of data, to make them consistent with the ontology, but more importantly to make them useful for the later linking.

This is done by:

1) Placing the information on bi-lateral trade flows in their respective market activities (for each of the 169 products, there are 49*49 bi-lateral markets, many of which will be empty (having no flows), and may therefore be ignored). This takes care of the disaggregated import data of the industries.

2) Aggregating the disaggregated import data of the industries, so that each industry only have 169 imported products, not the current 49*169 (since that information is already present in the above bi-lateral trade data).

This way of organising the Exiobase import preserves all data intact, and now in a more meaningful format that allows use by any relevant product system algorithm.

Of course, this transformation is completely transparent and an alternative could be to make an Exiobase ontology term for this "exiobase:import origin" and use this for importing the Exiobase data to RDF with this term attached and then do the "stripping" to BONSAI ontology in RDF. However, this would create precedence for making RDF ontologies for all other strange data formats that poeople wish to provide and making RDF converters for these. I do not think that is a road that we would want to go down. The whole purpose of the BONSAI ontology is to be lean and nevertheless complete enough to allow loss-less import of all the different kind of data people may wish to provide.

I hope this clarifies the reason for staying with the BONSAI ontology on this point and to adapt the import so that loss-less imprt of Exiobase nevertheless is possible.

Best regards

Bo

Den 2019-11-07 kl. 12.28 skrev Chris Mutel:
Sorry, I don't want to be a pain in the ass, but I think we are going
a bit down the raod of Stalin's ideological purity here... In an ideal
world, we would have separate sources of trade data, and could ignore
the EXIOBASE trade "assumptions"; but in this ideal world, we could
also take the SU tables directly from each country. EXIOBASE has the
trade information, and it is balanced. We need this trade data, and
don't really want to start a whole new project to get it from another
source (and clean it!). Bo says that "In a true SUT, the flows enter
and leave an activity but do not yet have information on their origin
and destination," but EXIOBASE is not just a SUT, it is also trade
data.

"The EXIOBASE SUT is overspecified in this sense that it already has
interpreted the information in the trade statistics in a specific
(attributional) way. This error should not be imported into the BONSAI
implementation, which should leave the user free to link SUT
activities with different linking algorithms." But we are free to
(re-)link SUT activities with different linking algorithms, even if we
import this data! All data is BONSAI are factual claims that we can
use or ignore as we wish.

We go here to a fundamental decision for the entire project, namely:
Should we let our collective or individual biases lead to data
modification **before** it enters the system? It was my impression
that our consensus decision from the hackathon was that we do not
alter or delete data before it enters the system, unless such
modification would never be controversial in any way (i.e. unit
conversions or changing labels in cases where there is zero
ambiguity). Did this change? I don't accept that it changed in a
comment in a Github issue where two people reported that they
discussed something offline.

On Wed, 6 Nov 2019 at 13:27, Matteo Lissandrini (AAU) <matteo@...> wrote:

Will this require to write from scratch the Exiobase RDF converter? do I understand correctly or is this about some other data?

But we need to update this software anyway to a) make it a proper
installable package, b) follow the URI schema that we are now using,
and c) fill in all the "TODO"s in the code.
Ok, I had the task to update this script for the USE table, as per other email, but then I'm blocked until I have the new output.

This means that for now, the published data is only for the supply table.

We will need to re-sync for completing this work.

Thanks,
Matteo



--


 

Thanks Bo, this is clear and (at least in my opinion) completely consistent with all group decisions. Maybe I just missed it - entirely possible! - but the issue discussions did not seem to have this clear framework. I will quote from this liberally when updating the software documentation.

If I understand correctly, we take the classic activity (maybe transforming activity :) and market approach, in that:
- Activities consume from and produce to markets
- All trade is between markets

The EXIOBASE importer will then need to create triples:
For each activity:
  In each place:
    For each flow object:
      A supply flow to the national market, if non-zero
      A use flow from the national market, if non-zero

For each place alpha
  For each other place beta
    For each flow object
      A trade flow from the national market in alpha to the national market in beta (trade volume is the sum of data in EXIOBASE)

The EXIOBASE RDF URI creator will need to create:

For each activity
  In each place
    An activity
    A market activity (currently missing, AFAICT)
For each flow object
  A flow object
For each place
  A place
  For every other place
    A trade activity (does trade need to be flow-object specific? In the future, does it need to be specific to transport mode?) (currently missing, AFAICT)

One minor technical question - do we take trade data from the supply or use table? From my uninformed perspective, I would expect this data to be the same in both tables, but I don't underestimate the ability of data providers to surprise me anymore!

Comments and clarifications welcome!

-Chris


On Thu, 7 Nov 2019 at 15:01, Bo Weidema <bo.weidema@...> wrote:

Dear Chris,

The issue is not about interpreting or deleting data, but simply to make the import of it consistent with our ontology. The Exiobase SUT data includes two kinds of data 1) inputs and outputs of industries ("production activities") 2) inputs and outputs of bi-lateral trade ("market activities"). Both kinds of flow data being SUT data, they do not have location as a property, i.e. the input flows are not linked to an origin, and the output flows are not linked to a destination. This linking is what happens when we produce the Direct Requirement Matrix using the product system algorithms on the SUT.

Since the Exiobase SUT has a convention of integrating the bi-lateral market data in the data for each industry, our import algorithm needs to separate these two kinds of data, to make them consistent with the ontology, but more importantly to make them useful for the later linking.

This is done by:

1) Placing the information on bi-lateral trade flows in their respective market activities (for each of the 169 products, there are 49*49 bi-lateral markets, many of which will be empty (having no flows), and may therefore be ignored). This takes care of the disaggregated import data of the industries.

2) Aggregating the disaggregated import data of the industries, so that each industry only have 169 imported products, not the current 49*169 (since that information is already present in the above bi-lateral trade data).

This way of organising the Exiobase import preserves all data intact, and now in a more meaningful format that allows use by any relevant product system algorithm.

Of course, this transformation is completely transparent and an alternative could be to make an Exiobase ontology term for this "exiobase:import origin" and use this for importing the Exiobase data to RDF with this term attached and then do the "stripping" to BONSAI ontology in RDF. However, this would create precedence for making RDF ontologies for all other strange data formats that poeople wish to provide and making RDF converters for these. I do not think that is a road that we would want to go down. The whole purpose of the BONSAI ontology is to be lean and nevertheless complete enough to allow loss-less import of all the different kind of data people may wish to provide.

I hope this clarifies the reason for staying with the BONSAI ontology on this point and to adapt the import so that loss-less imprt of Exiobase nevertheless is possible.

Best regards

Bo

Den 2019-11-07 kl. 12.28 skrev Chris Mutel:
Sorry, I don't want to be a pain in the ass, but I think we are going
a bit down the raod of Stalin's ideological purity here... In an ideal
world, we would have separate sources of trade data, and could ignore
the EXIOBASE trade "assumptions"; but in this ideal world, we could
also take the SU tables directly from each country. EXIOBASE has the
trade information, and it is balanced. We need this trade data, and
don't really want to start a whole new project to get it from another
source (and clean it!). Bo says that "In a true SUT, the flows enter
and leave an activity but do not yet have information on their origin
and destination," but EXIOBASE is not just a SUT, it is also trade
data.

"The EXIOBASE SUT is overspecified in this sense that it already has
interpreted the information in the trade statistics in a specific
(attributional) way. This error should not be imported into the BONSAI
implementation, which should leave the user free to link SUT
activities with different linking algorithms." But we are free to
(re-)link SUT activities with different linking algorithms, even if we
import this data! All data is BONSAI are factual claims that we can
use or ignore as we wish.

We go here to a fundamental decision for the entire project, namely:
Should we let our collective or individual biases lead to data
modification **before** it enters the system? It was my impression
that our consensus decision from the hackathon was that we do not
alter or delete data before it enters the system, unless such
modification would never be controversial in any way (i.e. unit
conversions or changing labels in cases where there is zero
ambiguity). Did this change? I don't accept that it changed in a
comment in a Github issue where two people reported that they
discussed something offline.

On Wed, 6 Nov 2019 at 13:27, Matteo Lissandrini (AAU) <matteo@...> wrote:

        
Will this require to write from scratch the Exiobase RDF converter? do I understand correctly or is this about some other data?

        
But we need to update this software anyway to a) make it a proper
installable package, b) follow the URI schema that we are now using,
and c) fill in all the "TODO"s in the code.
Ok, I had the task to update this script for the USE table, as per other email, but then I'm blocked until I have the new output.

This means that for now, the published data is only for the supply table.

We will need to re-sync for completing this work.

Thanks,
Matteo


    
--



--
############################
Chris Mutel
Technology Assessment Group, LEA
Paul Scherrer Institut
OHSA D22
5232 Villigen PSI
Switzerland
http://chris.mutel.org
Telefon: +41 56 310 5787
############################