Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sum results in concatenation rather than addition #166

Open
2 tasks
jbothma opened this issue May 1, 2019 · 0 comments
Open
2 tasks

sum results in concatenation rather than addition #166

jbothma opened this issue May 1, 2019 · 0 comments

Comments

@jbothma
Copy link
Contributor

jbothma commented May 1, 2019

In order to submit an issue, please ensure you can check the following. Thanks!

  • Declare which version of Python you are using (python --version)
  • Declare which operating system you are using
tableschema-sql==0.10.4
tabulate==0.8.2
tabulator==1.19.0
dataflows==0.0.32
datapackage==1.5.1
datapackage-pipelines==2.0.0

when I run a pipeline including the following steps, excluding the string_to_int processor, I get a cast error at the end of the pipeline because the value 100000-15297008027871520 is not an integer. It looks like the values from the csv are strings, and I suspect the sum operation does += resulting in concatenation.

    - run: load
      parameters:
        from: 'http://datastore.openspending.org/b9d2af843f3a7ca223eea07fb608e62a/estimates-of-provincial-expenditure-2018-19-uploaded-2019-05-01t1328/final/data/epre-2018-19.csv'
      cache: true

      # because otherwise string "addition" happens which looks like
      # '7871520' + '9700802' + '-152' + '100000' = '100000-15297008027871520'
    - run: string_to_int
      parameters:
        column_name: 'value'

    - run: filter
      parameters:
        resources: epre-2018-19
        in:
          - finyear: '2018'

    - run: delete_fields
      parameters:
        resources: epre-2018-19
        fields:
          - voteno
          - econ5
          - sprogno
          - subprogramme

    - run: join
      parameters:
        source:
          name: epre-2018-19
          key:
            - "province"
            - "department"
            - "progno"
            - "programme"
            - "fg1"
            - "fg2"
            - "econ1"
            - "econ2"
            - "econ3"
            - "econ4"
            - "finyear"
            - "budget_phase"
          delete: yes
        target:
          name: epre-2018-19
          key: null
        fields:
          "province": {}
          "department": {}
          "progno": {}
          "programme": {}
          "fg1": {}
          "fg2": {}
          "econ1": {}
          "econ2": {}
          "econ3": {}
          "econ4": {}
          "finyear": {}
          "budget_phase": {}
          "value":
            aggregate: sum
        full: true
| ERROR   :ROW: {'Amount Kind': 'Total', 'Budget Phase': 'Main appropriation', 'Department': 'Education', 'Econ1': 'Capital', 'Econ2': 'Payments for capital assets', 'Econ3': 'Machinery and equipment', 'Econ4': 'Other machinery and equipment', 'Financial Year': 2018, 'FunctionGroup1': 'Learning and culture', 'FunctionGroup2': 'Basic education', 'Government': 'Eastern Cape', 'Programme': 'Public Ordinary School Education', 'Programme Number': 2, 'Value': '100000-15297008027871520'}

Screenshot_2019-05-01_15-02-15

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant