Duplicate rows in the source

Last Post 01 Jun 2011 11:59 AM by PaulMcKibben. 1 Replies.
AddThis - Bookmarking and Sharing Button
Author Messages
New Member
New Member

01 Jun 2011 08:05 AM
I have a situation where the source is a file dump from a mainframe, and I have duplicate rows in the source. 

I have loaded it into a work table that has no keys or RI.

When I go to load it into the final table, I only can load a portion of the rows in the work table.  The package is performing like the first time that it hits a duplicate row, it is sending the rest of the rows to my error table.

I ultimately want to load only unique records into the final table.  I have no idea how to though.

Any assistance would be greatly appreciated.

Thank you.
New Member
New Member

01 Jun 2011 11:59 AM
I found the answer which is really simple.

After loading the work table, add an Exeute SQL Task with the following code:

INSERT INTO dbo.tbl1
SELECT DISTINCT user_id, access_control_cd, group_id, division_id, product_ind, network_ind
FROM work_db.dbo.work_tbl1

Acceptable Use Policy