Record duplication?

I have what appears to be duplicated records coming into my Birst data. I'm using 2 SQL query based objects (Labor + Employees) to bring in the data, and when I copy the SQL statement into another utility and run it the data is NOT duplicating. I checked the keys on the Labor table and I have them set to everything except the measurables.

I'm at a loss at this point as to how to further diagnose the issue. Does anyone have some ideas?

(I'm a Birst Noob, Sorry in advance!)

Find more posts tagged with

Comments

Legacy Contributor

Hi, we have been getting this a lot when we started using Birst and we noticed that when changing the refresh to full it resolved the issue. Also you might want to execute the delete all job first to delete all data in that table and change the refresh to full load and see the result. This should take care of the issue.

Hopefully this works for you.

Diane McElhiney

While a full refresh may correct the issue, it is not an acceptable, sustainable path especially if the data volume being extracted grows.

With that stated, a screenshot doesn't really help in determining what is occurring (it is blurry when enlarged). Please paste the BQL from the Scripted Source SELECT section and the Script section from the script where you feel this duplication is occurring. If you add a WHERE clause in the Scripted Source to select only the row(s) you believe are duplicating, in the Preview are two rows displayed or one?

Also, what is the snapshot/transaction policy? Incremental refresh must have the unique key defined to find the existing in order to delete prior to insert. If no key columns defined, the Incremental refresh policy is then an append which would produce duplicate data as the Birst engine interprets the data as being mutual exclusive.

You can review the processing log from Monitoring for the job where you state duplicates are occurring and scan for the script being processed to see the DML SQL statements used to populate the measure table.

Legacy Contributor

Hi, Ferno. Looking at your report I see both Payroll and Orders. You may be seeing duplication because a given employee has multiple orders, or multiple paychecks. Essentially, there's a many to many relationship between orders and payroll, so if you're using them together in a report they will need to be modeled carefully. I'm not sure you can come to the conclusion that this data is duplicated based on this report.

You probably need to look at Space Management --> View Processed Data and look at the data in each of the fact tables that are used in this report to truly know if data is being duplicated when loaded into Birst or if your model doesn't really make sense, resulting in duplicate rows when tables are joined together in a report.

Legacy Contributor

Hi Diane,

I did a full refresh and it seems the issue has been resolved. This may be an ok solution, considering I intend to only bring in 12 months of data at a time (should then reset at month-end).

I have key columns defined on the labor master, anything that isn't a measurable.

Diane McElhiney

@a-boninsegna

Just another point. The key columns defined on an Incremental Refresh policy are those columns which define the unique combination for Birst engine to delete a row. If you've defined too many columns as the key, you may be restricting the 'uniqueness', especially if one of the columns is a date which can change from the time the row was initially loaded vs. subsequent load.

If you feel the data volume month to month is around the same and the percentage of rows changed is a majority of that data being extracted/published, then a full refresh is the best direction.

If you wish to explore the snapshot/transaction policy further, you can communication with your Infor contact and ask for them to inquire the TSM Specialty team.

CEB Call-20210318_110330-Meeting Recording.mp4

Copyright © 2025 Infor. All rights reserved.