Showing posts with label back. Show all posts
Showing posts with label back. Show all posts

Friday, March 23, 2012

problems with replication while rolling back nested transactions

Replication appears to be reading rolled back transactions. We are seeing
rows on our subscriber that do not exist on our publisher. The data on the
publisher is what is expected, the rolled back rows appearing on the
subscriber obviously are not.
Here's the basic structure of my two stored procs. They are done this way
because, depending on business rules, I can call the inner proc by itself or
from the outer.
sp_outer
{ basics of the outer
SET IMPLICIT_TRANSACTIONS OFF
Begin Transaction
Save Transaction Group
While more deals
Call sp_inner
{ basics of the inner
SET IMPLICIT_TRANSACTIONS OFF
Begin Transaction
Save Transaction Individual
work...
If Fail ROLLBACK Individual
If Pass COMMIT
}
End
If Any Fail in inner ROLLBACK Group outer
Else COMMIT outer
}
Problem is occuring when outer transaction rolls back. From on the
sqlserver docs "Committing inner transactions is ignored by Microsoft? SQL
Server. The transaction is either committed or rolled back based on the
action taken at the end of the outermost transaction. If the outer
transaction is committed, the inner nested transactions are also committed.
If the outer transaction is rolled back, then all inner transactions are also
rolled back, regardless of whether or not the inner transactions were
individually committed."
Again, what we see on our publisher is as expected. Subscribers having
different rows are the issue.
The chart at the bottom shows different ordering of the rollbacks and
commits and establishes a pattern of the problem occurs with commited
transactions prior to or between two rolled back transactions.
This sure looks like a replication bug as the publisher behaves as expected.
What do you think?
Exists on PublisherExists on Subscriber
Begin Outer
Rollback InnerNoNo
Commit InnerNoYes
Commit InnerNoYes
Rollback InnerNoNo
Rollback Outer
Begin Outer
Commit InnerNoYes
Rollback InnerNoNo
Commit InnerNoYes
Rollback InnerNoNo
Rollback Outer
Begin Outer
Rollback InnerNoNo
Commit InnerNoYes
Rollback InnerNoNo
Commit InnerNoNo
Rollback Outer
Begin Outer
Rollback InnerNoNo
Rollback InnerNoNo
Commit InnerNoNo
Commit InnerNoNo
Rollback Outer
Begin Outer
Commit InnerNoNo
Commit InnerNoNo
Commit InnerNoNo
Rollback InnerNoNo
Rollback Outer
Begin Outer
Rollback InnerNoNo
Commit InnerNoNo
Commit InnerNoNo
Commit InnerNoNo
Rollback Outer
TPerry,
I admire your methodicalness (if this is a proper word) and agree that this
looks confusing
If we compare

> Begin Outer
> Rollback Inner No No
> Commit Inner No Yes
> Commit Inner No Yes
> Rollback Inner No No
> Rollback Outer
and

> Begin Outer
> Rollback Inner No No
> Commit Inner No No
> Commit Inner No No
> Commit Inner No No
> Rollback Outer
Why should the first and second commits should be different? Interesting!
I can't test all this right now, but one thing I did test in the past was
that setting the transaction isolation level as serializable and setting
xact_abort on before running the transactions altered the behaviour; if code
in a stored procedure was rolled back on the publisher, it wasn't sent to
the subscriber. Please can you test your scenario under these conditions to
see if this fixes your issue.
Rgds,
Paul Ibison (SQL Server MVP)
[vbcol=seagreen]
|||If a transaction is logged, it will be replicated.
If the transaction is not committed the log reader can't read it, and hence
it will not be replicated.
If you can isolate a sequence of transactions that cause this phenomena you
could use sp_browsereplcmds to see if they make it to the distribution
database. If so they will be replicated.
It sounds like to me there is a logical error somewhere in your proc.
"tperry" <tperry@.discussions.microsoft.com> wrote in message
news:D2DF5BE8-66E0-45E6-891D-DD61BD23EBD3@.microsoft.com...
> Replication appears to be reading rolled back transactions. We are seeing
> rows on our subscriber that do not exist on our publisher. The data on
> the
> publisher is what is expected, the rolled back rows appearing on the
> subscriber obviously are not.
> Here's the basic structure of my two stored procs. They are done this way
> because, depending on business rules, I can call the inner proc by itself
> or
> from the outer.
> sp_outer
> { basics of the outer
> SET IMPLICIT_TRANSACTIONS OFF
> Begin Transaction
> Save Transaction Group
> While more deals
> Call sp_inner
> { basics of the inner
> SET IMPLICIT_TRANSACTIONS OFF
> Begin Transaction
> Save Transaction Individual
> work...
> If Fail ROLLBACK Individual
> If Pass COMMIT
> }
> End
> If Any Fail in inner ROLLBACK Group outer
> Else COMMIT outer
> }
> Problem is occuring when outer transaction rolls back. From on the
> sqlserver docs "Committing inner transactions is ignored by Microsoft?
> SQL
> Server. The transaction is either committed or rolled back based on the
> action taken at the end of the outermost transaction. If the outer
> transaction is committed, the inner nested transactions are also
> committed.
> If the outer transaction is rolled back, then all inner transactions are
> also
> rolled back, regardless of whether or not the inner transactions were
> individually committed."
> Again, what we see on our publisher is as expected. Subscribers having
> different rows are the issue.
> The chart at the bottom shows different ordering of the rollbacks and
> commits and establishes a pattern of the problem occurs with commited
> transactions prior to or between two rolled back transactions.
> This sure looks like a replication bug as the publisher behaves as
> expected.
> What do you think?
>
> Exists on Publisher Exists on Subscriber
> Begin Outer
> Rollback Inner No No
> Commit Inner No Yes
> Commit Inner No Yes
> Rollback Inner No No
> Rollback Outer
> Begin Outer
> Commit Inner No Yes
> Rollback Inner No No
> Commit Inner No Yes
> Rollback Inner No No
> Rollback Outer
> Begin Outer
> Rollback Inner No No
> Commit Inner No Yes
> Rollback Inner No No
> Commit Inner No No
> Rollback Outer
> Begin Outer
> Rollback Inner No No
> Rollback Inner No No
> Commit Inner No No
> Commit Inner No No
> Rollback Outer
> Begin Outer
> Commit Inner No No
> Commit Inner No No
> Commit Inner No No
> Rollback Inner No No
> Rollback Outer
> Begin Outer
> Rollback Inner No No
> Commit Inner No No
> Commit Inner No No
> Commit Inner No No
> Rollback Outer
>

Tuesday, March 20, 2012

Problems with mirroring

I have the three server ready to go. The prinicpal is set, the mirror
is set. I have a back up of the prinicipal restored on the mirror and
then I have a third server that acts as the witness. The problem is I
am getting an error message. I am getting this error message:
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
\\\\\\\\\\\\\
TITLE: Database Properties
--
An error occurred while starting mirroring.
ADDITIONAL INFORMATION:
Alter failed for Database 'SQLServer'. (Microsoft.SqlServer.Smo)
For help, click:
http://go.microsoft.com/fwlink?Prod...se&LinkId=20476
An exception occurred while executing a Transact-SQL statement or
batch. (Microsoft.SqlServer.ConnectionInfo)
The server network address "TCP://xxx.xxx.org:5022" can not be reached
or does not exist. Check the network address name and that the ports
for the local and remote endpoints are operational. (Microsoft SQL
Server, Error: 1418)
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
\\\\\\\\\\\\\\\\\\\\\\\\\\\
Any ideas would most helpful.
-ThanasCan you show your scripts that you used to create your mirrors?
Did you open the ports on your firewall?
Thanas wrote:
> I have the three server ready to go. The prinicpal is set, the mirror
> is set. I have a back up of the prinicipal restored on the mirror and
> then I have a third server that acts as the witness. The problem is I
> am getting an error message. I am getting this error message:
> \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
\\\\\\\\\\\\\\\
> TITLE: Database Properties
> --
> An error occurred while starting mirroring.
> --
> ADDITIONAL INFORMATION:
> Alter failed for Database 'SQLServer'. (Microsoft.SqlServer.Smo)
> For help, click:
> http://go.microsoft.com/fwlink?Prod...se&LinkId=20476
> --
> An exception occurred while executing a Transact-SQL statement or
> batch. (Microsoft.SqlServer.ConnectionInfo)
> --
> The server network address "TCP://xxx.xxx.org:5022" can not be reached
> or does not exist. Check the network address name and that the ports
> for the local and remote endpoints are operational. (Microsoft SQL
> Server, Error: 1418)
> \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
>
> Any ideas would most helpful.
> -Thanas|||Used Management Studio to generate the mirror. No firewall up, all
internal. There is not even a windows firewall.
PSPDBA wrote:[vbcol=seagreen]
> Can you show your scripts that you used to create your mirrors?
> Did you open the ports on your firewall?
>
> Thanas wrote:|||Try using TCP://ServerName:5022 rather than the full path.
Thanas wrote:[vbcol=seagreen]
> Used Management Studio to generate the mirror. No firewall up, all
> internal. There is not even a windows firewall.
> PSPDBA wrote:|||Read step 2 in the section "Troubleshooting" in the following document.
http://www.microsoft.com/technet/pr...5/dbmirror.mspx
If your services are running as LocalSystem, it will not work. Try
changing to an account with the appropriate privileges. I changed to a
domian account and my problem went away.
Good Luck
Jamie|||Still having trouble. Not really sure why anymore. here is the error
message that I got. Of course I have "x" out my server information for
obvious reason ...
The witness is a SQL Server Express. It is up to date as far as
updates goes.
Thanks for your help and patience everyone.
-Thanas
TITLE: Database Properties
--
An error occurred while starting mirroring.
ADDITIONAL INFORMATION:
Alter failed for Database 'xxxxSQLServerDev'.
(Microsoft.SqlServer.Smo)
For help, click:
http://go.microsoft.com/fwlink?Prod...se&LinkId=20476
An exception occurred while executing a Transact-SQL statement or
batch. (Microsoft.SqlServer.ConnectionInfo)
The ALTER DATABASE command could not be sent to the remote server
instance 'TCP://xxx.xxx.org:5022'. The database mirroring configuration
was not changed. Verify that the server is connected, and try again.
(Microsoft SQL Server, Error: 1456)
For help, click:
http://go.microsoft.com/fwlink?Prod...56&LinkId=20476
BUTTONS:
OK
--
jamie@.encompassgroup.ca wrote:
> Read step 2 in the section "Troubleshooting" in the following document.
> http://www.microsoft.com/technet/pr...5/dbmirror.mspx
> If your services are running as LocalSystem, it will not work. Try
> changing to an account with the appropriate privileges. I changed to a
> domian account and my problem went away.
> Good Luck
> Jamie

Problems with mirroring

I have the three server ready to go. The prinicpal is set, the mirror
is set. I have a back up of the prinicipal restored on the mirror and
then I have a third server that acts as the witness. The problem is I
am getting an error message. I am getting this error message:
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
TITLE: Database Properties
--
An error occurred while starting mirroring.
--
ADDITIONAL INFORMATION:
Alter failed for Database 'SQLServer'. (Microsoft.SqlServer.Smo)
For help, click:
http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=9.00.2047.00&EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&EvtID=Alter+Database&LinkId=20476
--
An exception occurred while executing a Transact-SQL statement or
batch. (Microsoft.SqlServer.ConnectionInfo)
--
The server network address "TCP://xxx.xxx.org:5022" can not be reached
or does not exist. Check the network address name and that the ports
for the local and remote endpoints are operational. (Microsoft SQL
Server, Error: 1418)
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
Any ideas would most helpful.
-ThanasCan you show your scripts that you used to create your mirrors?
Did you open the ports on your firewall?
Thanas wrote:
> I have the three server ready to go. The prinicpal is set, the mirror
> is set. I have a back up of the prinicipal restored on the mirror and
> then I have a third server that acts as the witness. The problem is I
> am getting an error message. I am getting this error message:
> \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
> TITLE: Database Properties
> --
> An error occurred while starting mirroring.
> --
> ADDITIONAL INFORMATION:
> Alter failed for Database 'SQLServer'. (Microsoft.SqlServer.Smo)
> For help, click:
> http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=9.00.2047.00&EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&EvtID=Alter+Database&LinkId=20476
> --
> An exception occurred while executing a Transact-SQL statement or
> batch. (Microsoft.SqlServer.ConnectionInfo)
> --
> The server network address "TCP://xxx.xxx.org:5022" can not be reached
> or does not exist. Check the network address name and that the ports
> for the local and remote endpoints are operational. (Microsoft SQL
> Server, Error: 1418)
> \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
>
> Any ideas would most helpful.
> -Thanas|||Used Management Studio to generate the mirror. No firewall up, all
internal. There is not even a windows firewall.
PSPDBA wrote:
> Can you show your scripts that you used to create your mirrors?
> Did you open the ports on your firewall?
>
> Thanas wrote:
> > I have the three server ready to go. The prinicpal is set, the mirror
> > is set. I have a back up of the prinicipal restored on the mirror and
> > then I have a third server that acts as the witness. The problem is I
> > am getting an error message. I am getting this error message:
> >
> > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
> > TITLE: Database Properties
> > --
> >
> > An error occurred while starting mirroring.
> >
> > --
> > ADDITIONAL INFORMATION:
> >
> > Alter failed for Database 'SQLServer'. (Microsoft.SqlServer.Smo)
> >
> > For help, click:
> > http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=9.00.2047.00&EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&EvtID=Alter+Database&LinkId=20476
> >
> > --
> >
> > An exception occurred while executing a Transact-SQL statement or
> > batch. (Microsoft.SqlServer.ConnectionInfo)
> >
> > --
> >
> > The server network address "TCP://xxx.xxx.org:5022" can not be reached
> > or does not exist. Check the network address name and that the ports
> > for the local and remote endpoints are operational. (Microsoft SQL
> > Server, Error: 1418)
> > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
> >
> >
> > Any ideas would most helpful.
> > -Thanas|||Try using TCP://ServerName:5022 rather than the full path.
Thanas wrote:
> Used Management Studio to generate the mirror. No firewall up, all
> internal. There is not even a windows firewall.
> PSPDBA wrote:
> > Can you show your scripts that you used to create your mirrors?
> >
> > Did you open the ports on your firewall?
> >
> >
> > Thanas wrote:
> > > I have the three server ready to go. The prinicpal is set, the mirror
> > > is set. I have a back up of the prinicipal restored on the mirror and
> > > then I have a third server that acts as the witness. The problem is I
> > > am getting an error message. I am getting this error message:
> > >
> > > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
> > > TITLE: Database Properties
> > > --
> > >
> > > An error occurred while starting mirroring.
> > >
> > > --
> > > ADDITIONAL INFORMATION:
> > >
> > > Alter failed for Database 'SQLServer'. (Microsoft.SqlServer.Smo)
> > >
> > > For help, click:
> > > http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=9.00.2047.00&EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&EvtID=Alter+Database&LinkId=20476
> > >
> > > --
> > >
> > > An exception occurred while executing a Transact-SQL statement or
> > > batch. (Microsoft.SqlServer.ConnectionInfo)
> > >
> > > --
> > >
> > > The server network address "TCP://xxx.xxx.org:5022" can not be reached
> > > or does not exist. Check the network address name and that the ports
> > > for the local and remote endpoints are operational. (Microsoft SQL
> > > Server, Error: 1418)
> > > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
> > >
> > >
> > > Any ideas would most helpful.
> > > -Thanas|||Read step 2 in the section "Troubleshooting" in the following document.
http://www.microsoft.com/technet/prodtechnol/sql/2005/dbmirror.mspx
If your services are running as LocalSystem, it will not work. Try
changing to an account with the appropriate privileges. I changed to a
domian account and my problem went away.
Good Luck
Jamie|||Still having trouble. Not really sure why anymore. here is the error
message that I got. Of course I have "x" out my server information for
obvious reason :)...
The witness is a SQL Server Express. It is up to date as far as
updates goes.
Thanks for your help and patience everyone.
-Thanas
TITLE: Database Properties
--
An error occurred while starting mirroring.
--
ADDITIONAL INFORMATION:
Alter failed for Database 'xxxxSQLServerDev'.
(Microsoft.SqlServer.Smo)
For help, click:
http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=9.00.2047.00&EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&EvtID=Alter+Database&LinkId=20476
--
An exception occurred while executing a Transact-SQL statement or
batch. (Microsoft.SqlServer.ConnectionInfo)
--
The ALTER DATABASE command could not be sent to the remote server
instance 'TCP://xxx.xxx.org:5022'. The database mirroring configuration
was not changed. Verify that the server is connected, and try again.
(Microsoft SQL Server, Error: 1456)
For help, click:
http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=09.00.2047&EvtSrc=MSSQLServer&EvtID=1456&LinkId=20476
--
BUTTONS:
OK
--
jamie@.encompassgroup.ca wrote:
> Read step 2 in the section "Troubleshooting" in the following document.
> http://www.microsoft.com/technet/prodtechnol/sql/2005/dbmirror.mspx
> If your services are running as LocalSystem, it will not work. Try
> changing to an account with the appropriate privileges. I changed to a
> domian account and my problem went away.
> Good Luck
> Jamie

Friday, March 9, 2012

problems with diff backup

hello.
my server sql2000 has this backup schema: monday-full and other days
diff backup.
But 2weeks back diff backups become too large -cca 1.3 GB. previous
size of diff backups was about 300mb. every diff backup in week backup
schema is incerasing from 1.3GB to 1,4GB. largest table in database is
about 500MB large. so I can't identify why about 1.3 GB is backuped
every day. Can anybody help me ?
How Diff backup stores changes ? by tables or by rows ? Thank for
answer.
thansk .
Perhaps your indexes are fragmented. Try running DBCC INDEXDEFRAG or DBCC
DBREINDEX. Then see if the problem persists.
Tom
Thomas A. Moreau, BSc, PhD, MCSE, MCDBA
SQL Server MVP
Columnist, SQL Server Professional
Toronto, ON Canada
www.pinnaclepublishing.com/sql
"oto pakozdy" <pakozdy@.pobox.sk> wrote in message
news:fa2dfe40.0409210421.7ea75192@.posting.google.c om...
hello.
my server sql2000 has this backup schema: monday-full and other days
diff backup.
But 2weeks back diff backups become too large -cca 1.3 GB. previous
size of diff backups was about 300mb. every diff backup in week backup
schema is incerasing from 1.3GB to 1,4GB. largest table in database is
about 500MB large. so I can't identify why about 1.3 GB is backuped
every day. Can anybody help me ?
How Diff backup stores changes ? by tables or by rows ? Thank for
answer.
thansk .

problems with diff backup

hello.
my server sql2000 has this backup schema: monday-full and other days
diff backup.
But 2weeks back diff backups become too large -cca 1.3 GB. previous
size of diff backups was about 300mb. every diff backup in week backup
schema is incerasing from 1.3GB to 1,4GB. largest table in database is
about 500MB large. so I can't identify why about 1.3 GB is backuped
every day. Can anybody help me ?
How Diff backup stores changes ? by tables or by rows ? Thank for
answer.
thansk .Perhaps your indexes are fragmented. Try running DBCC INDEXDEFRAG or DBCC
DBREINDEX. Then see if the problem persists.
--
Tom
---
Thomas A. Moreau, BSc, PhD, MCSE, MCDBA
SQL Server MVP
Columnist, SQL Server Professional
Toronto, ON Canada
www.pinnaclepublishing.com/sql
"oto pakozdy" <pakozdy@.pobox.sk> wrote in message
news:fa2dfe40.0409210421.7ea75192@.posting.google.com...
hello.
my server sql2000 has this backup schema: monday-full and other days
diff backup.
But 2weeks back diff backups become too large -cca 1.3 GB. previous
size of diff backups was about 300mb. every diff backup in week backup
schema is incerasing from 1.3GB to 1,4GB. largest table in database is
about 500MB large. so I can't identify why about 1.3 GB is backuped
every day. Can anybody help me ?
How Diff backup stores changes ? by tables or by rows ? Thank for
answer.
thansk .

Monday, February 20, 2012

Problems Using Temporary Tables

I am declaring a temp table in control flow via Execute SQL task. I then want to use that table as a source in a data task, keep it alive going back to control flow and use it as a destination in another data task following. I am having trouble just getting the first data flow to access that temp table.

In my Execute SQL task, I'm creating a simple temp table. After executing that task, I can head to the data task and change my OLE DB Source to point to that temp table, but I recieve an error about an invalid table name when I attempt to map columns or hit "OK."

Is there something else I need to do to use that temp table as a source or can I only reference it through SQL queries?

Thank you.

I have found them to be unreliable and problematic to use in SSIS. The meta-data issues during dev. for example are a real pain, and the difficulty of controlling the lifetime of the temp tables is another.

I am now using table variables where I previously used temp tables. Table variables are typically faster for small row counts but slower with greater row counts. Unfortunately, I am using them for greater row counts and my sql statement went from 5 1/2 minutes to 8 1/2 minutes for about 250,000 rows.

|||How would I go about doing that though? I would like to try it out even though I'm aware of the limited success.
|||

Well, you've been warned :)

It's my understanding that the only thing you can do to SSIS to specifically facilitate the use of temp tables is to set the "retain same connection" property to "true" for the connection that is used to create the temp table, and then use that connection for all references to the table.

Ken

|||

Although you can use temporary tables in tempdb (#MyTempTable) by setting RetainSameConnection, you might prefer creating a temporary table in an ordinary database (MyTempTable), without setting RetainSameConnection:

Create the table ahead of time. Truncate it before using it on each invocation.

Cheers/Kristian

|||

kenambrose wrote:

Well, you've been warned :)

It's my understanding that the only thing you can do to SSIS to specifically facilitate the use of temp tables is to set the "retain same connection" property to "true" for the connection that is used to create the temp table, and then use that connection for all references to the table.

Ken

Right, I've seen this answer before, and actually did this. It allowed me to see the temp table after executing the task (as #TempTable_0283817, or something similar) but when I select it, I recieve an error.

I will reply later with the specific error in a few hours when I can get to the SQL box.
|||Here is the error message that pops up when I try to map columns to the temp table I chose from the drop down.

TITLE: Microsoft Visual Studio

Error at Data Flow Task [OLE DB Destination [340]]: An OLE DB error has occurred. Error code: 0x80040E37.

Error at Data Flow Task [OLE DB Destination [340]]: Opening a rowset for "[dbo].[#TestTable_000000002219]" failed. Check that the object exists in the database.

ADDITIONAL INFORMATION:

Exception from HRESULT: 0xC02020E8 (Microsoft.SqlServer.DTSPipelineWrap)

BUTTONS:

OK

|||

Yes that works of course but then the SSIS user context needs "create object" and "write" permissions on the source data server (unless you are willing to suffer the performance disaster of heterogenerous joins). Bad and/or impossible for most extraction scenarios.....

Ken

|||That's what the staging database is for in most extract scenarios, I find that even creating tempdb tables on the source systems are seldom OK.

Either way, by having an authorized user creating the table before SSIS runs, it's enough to give the SSIS context insert/delete rights.

Cheers/Kristian|||

Kristian Wedberg wrote:

That's what the staging database is for in most extract scenarios, I find that even creating tempdb tables on the source systems are seldom OK.

Either way, by having an authorized user creating the table before SSIS runs, it's enough to give the SSIS context insert/delete rights.

Cheers/Kristian

When you say "staging database" are you referring to an object in the SSIS package or is that just a general term?

Basically, what I want to do is run through the entire input file, validate it, and record any errors I find before inserting any DB records.
|||

In my use of temp tables and the OLE DB Source or Destination I have resorted to using the Advanced Editor where you can create the columns and mappings "by hand". An easy way to see how to set them up is to create a table in your dev db that looks like your temp table. Set up the source or destination using that table and map your columns. Then open up the Advanced Editor.

One thing I found with temp tables. Of course you know about the RetainSameConnection setting. I was using the same connection to create the temp table, fill it, then open it so that I could write to another table on the same connection. Where I write to another table on the same connection I had to create a new separate connection and use it to write to the other table. Otherwise it seemed the viability of the temp table was uncertain when I run a second parallel data flow task where that temp table would be used as well.

Put another way, I use one connection to create the temp table, fill it and read from it. I do not use that same connection to write or read from any other tables. This may not be effecient but it was the only way I could get things to work in parallel.

Hope this helps,
David Martin

|||We are loading data from several source systems simultaneously. Each system generates a set of identical files using the Raw Data File destination. All extracts share common load procedures that pick up the generated Raw Data files through an expression variable. It's a pretty slick way around temp tables and seems to work pretty fast. Would be nicer if there was a Recordset source and we didn't have to create the file on the server but for now there isn't one. You may want to look at that approach as a way around temporary tables.|||

Martin is absolutely right. Raw files are akin to temp tables.

He is correct that there is no recordset source (and with good reason) but you can build one yourself as explined here: http://blogs.conchango.com/jamiethomson/archive/2006/01/04/2540.aspx

You can't append to a recordset destination unfortunately. If you think that would be nice functionality to have then vote for it here: http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedbackid=6831c0b9-ef74-4132-a137-b073a950d3fd

-Jamie

|||I meant "staging database" as a general term, i.e. it's often useful for the SSIS part to have access to a database where it can write and read arbitrary tables, even if that database is different from both your source database and your target database.|||

<< we didn't have to create the file on the server >>

That's fine if:

a: Your package runs local to your source data (very rare I expect)

or

b: You do not need to reference temp data structures in the queries that populate the raw data files.

But when you need the functionality of temporary data structures when working with queries that reference remote source data, it's either heterogenerous joins (death to performance) or table variables.

Ken

Problems Using Temporary Tables

I am declaring a temp table in control flow via Execute SQL task. I then want to use that table as a source in a data task, keep it alive going back to control flow and use it as a destination in another data task following. I am having trouble just getting the first data flow to access that temp table.

In my Execute SQL task, I'm creating a simple temp table. After executing that task, I can head to the data task and change my OLE DB Source to point to that temp table, but I recieve an error about an invalid table name when I attempt to map columns or hit "OK."

Is there something else I need to do to use that temp table as a source or can I only reference it through SQL queries?

Thank you.

I have found them to be unreliable and problematic to use in SSIS. The meta-data issues during dev. for example are a real pain, and the difficulty of controlling the lifetime of the temp tables is another.

I am now using table variables where I previously used temp tables. Table variables are typically faster for small row counts but slower with greater row counts. Unfortunately, I am using them for greater row counts and my sql statement went from 5 1/2 minutes to 8 1/2 minutes for about 250,000 rows.

|||How would I go about doing that though? I would like to try it out even though I'm aware of the limited success.|||

Well, you've been warned :)

It's my understanding that the only thing you can do to SSIS to specifically facilitate the use of temp tables is to set the "retain same connection" property to "true" for the connection that is used to create the temp table, and then use that connection for all references to the table.

Ken

|||

Although you can use temporary tables in tempdb (#MyTempTable) by setting RetainSameConnection, you might prefer creating a temporary table in an ordinary database (MyTempTable), without setting RetainSameConnection:

Create the table ahead of time. Truncate it before using it on each invocation.

Cheers/Kristian

|||

kenambrose wrote:

Well, you've been warned :)

It's my understanding that the only thing you can do to SSIS to specifically facilitate the use of temp tables is to set the "retain same connection" property to "true" for the connection that is used to create the temp table, and then use that connection for all references to the table.

Ken

Right, I've seen this answer before, and actually did this. It allowed me to see the temp table after executing the task (as #TempTable_0283817, or something similar) but when I select it, I recieve an error.

I will reply later with the specific error in a few hours when I can get to the SQL box.|||Here is the error message that pops up when I try to map columns to the temp table I chose from the drop down.

TITLE: Microsoft Visual Studio

Error at Data Flow Task [OLE DB Destination [340]]: An OLE DB error has occurred. Error code: 0x80040E37.

Error at Data Flow Task [OLE DB Destination [340]]: Opening a rowset for "[dbo].[#TestTable_000000002219]" failed. Check that the object exists in the database.

ADDITIONAL INFORMATION:

Exception from HRESULT: 0xC02020E8 (Microsoft.SqlServer.DTSPipelineWrap)

BUTTONS:

OK|||

Yes that works of course but then the SSIS user context needs "create object" and "write" permissions on the source data server (unless you are willing to suffer the performance disaster of heterogenerous joins). Bad and/or impossible for most extraction scenarios.....

Ken

|||That's what the staging database is for in most extract scenarios, I find that even creating tempdb tables on the source systems are seldom OK.

Either way, by having an authorized user creating the table before SSIS runs, it's enough to give the SSIS context insert/delete rights.

Cheers/Kristian|||

Kristian Wedberg wrote:

That's what the staging database is for in most extract scenarios, I find that even creating tempdb tables on the source systems are seldom OK.

Either way, by having an authorized user creating the table before SSIS runs, it's enough to give the SSIS context insert/delete rights.

Cheers/Kristian

When you say "staging database" are you referring to an object in the SSIS package or is that just a general term?

Basically, what I want to do is run through the entire input file, validate it, and record any errors I find before inserting any DB records.|||

In my use of temp tables and the OLE DB Source or Destination I have resorted to using the Advanced Editor where you can create the columns and mappings "by hand". An easy way to see how to set them up is to create a table in your dev db that looks like your temp table. Set up the source or destination using that table and map your columns. Then open up the Advanced Editor.

One thing I found with temp tables. Of course you know about the RetainSameConnection setting. I was using the same connection to create the temp table, fill it, then open it so that I could write to another table on the same connection. Where I write to another table on the same connection I had to create a new separate connection and use it to write to the other table. Otherwise it seemed the viability of the temp table was uncertain when I run a second parallel data flow task where that temp table would be used as well.

Put another way, I use one connection to create the temp table, fill it and read from it. I do not use that same connection to write or read from any other tables. This may not be effecient but it was the only way I could get things to work in parallel.

Hope this helps,
David Martin

|||We are loading data from several source systems simultaneously. Each system generates a set of identical files using the Raw Data File destination. All extracts share common load procedures that pick up the generated Raw Data files through an expression variable. It's a pretty slick way around temp tables and seems to work pretty fast. Would be nicer if there was a Recordset source and we didn't have to create the file on the server but for now there isn't one. You may want to look at that approach as a way around temporary tables.|||

Martin is absolutely right. Raw files are akin to temp tables.

He is correct that there is no recordset source (and with good reason) but you can build one yourself as explined here: http://blogs.conchango.com/jamiethomson/archive/2006/01/04/2540.aspx

You can't append to a recordset destination unfortunately. If you think that would be nice functionality to have then vote for it here: http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedbackid=6831c0b9-ef74-4132-a137-b073a950d3fd

-Jamie

|||I meant "staging database" as a general term, i.e. it's often useful for the SSIS part to have access to a database where it can write and read arbitrary tables, even if that database is different from both your source database and your target database.|||

<< we didn't have to create the file on the server >>

That's fine if:

a: Your package runs local to your source data (very rare I expect)

or

b: You do not need to reference temp data structures in the queries that populate the raw data files.

But when you need the functionality of temporary data structures when working with queries that reference remote source data, it's either heterogenerous joins (death to performance) or table variables.

Ken

Problems Using Temporary Tables

I am declaring a temp table in control flow via Execute SQL task. I then want to use that table as a source in a data task, keep it alive going back to control flow and use it as a destination in another data task following. I am having trouble just getting the first data flow to access that temp table.

In my Execute SQL task, I'm creating a simple temp table. After executing that task, I can head to the data task and change my OLE DB Source to point to that temp table, but I recieve an error about an invalid table name when I attempt to map columns or hit "OK."

Is there something else I need to do to use that temp table as a source or can I only reference it through SQL queries?

Thank you.

I have found them to be unreliable and problematic to use in SSIS. The meta-data issues during dev. for example are a real pain, and the difficulty of controlling the lifetime of the temp tables is another.

I am now using table variables where I previously used temp tables. Table variables are typically faster for small row counts but slower with greater row counts. Unfortunately, I am using them for greater row counts and my sql statement went from 5 1/2 minutes to 8 1/2 minutes for about 250,000 rows.

|||How would I go about doing that though? I would like to try it out even though I'm aware of the limited success.|||

Well, you've been warned :)

It's my understanding that the only thing you can do to SSIS to specifically facilitate the use of temp tables is to set the "retain same connection" property to "true" for the connection that is used to create the temp table, and then use that connection for all references to the table.

Ken

|||

Although you can use temporary tables in tempdb (#MyTempTable) by setting RetainSameConnection, you might prefer creating a temporary table in an ordinary database (MyTempTable), without setting RetainSameConnection:

Create the table ahead of time. Truncate it before using it on each invocation.

Cheers/Kristian

|||

kenambrose wrote:

Well, you've been warned :)

It's my understanding that the only thing you can do to SSIS to specifically facilitate the use of temp tables is to set the "retain same connection" property to "true" for the connection that is used to create the temp table, and then use that connection for all references to the table.

Ken

Right, I've seen this answer before, and actually did this. It allowed me to see the temp table after executing the task (as #TempTable_0283817, or something similar) but when I select it, I recieve an error.

I will reply later with the specific error in a few hours when I can get to the SQL box.|||Here is the error message that pops up when I try to map columns to the temp table I chose from the drop down.

TITLE: Microsoft Visual Studio

Error at Data Flow Task [OLE DB Destination [340]]: An OLE DB error has occurred. Error code: 0x80040E37.

Error at Data Flow Task [OLE DB Destination [340]]: Opening a rowset for "[dbo].[#TestTable_000000002219]" failed. Check that the object exists in the database.

ADDITIONAL INFORMATION:

Exception from HRESULT: 0xC02020E8 (Microsoft.SqlServer.DTSPipelineWrap)

BUTTONS:

OK|||

Yes that works of course but then the SSIS user context needs "create object" and "write" permissions on the source data server (unless you are willing to suffer the performance disaster of heterogenerous joins). Bad and/or impossible for most extraction scenarios.....

Ken

|||That's what the staging database is for in most extract scenarios, I find that even creating tempdb tables on the source systems are seldom OK.

Either way, by having an authorized user creating the table before SSIS runs, it's enough to give the SSIS context insert/delete rights.

Cheers/Kristian|||

Kristian Wedberg wrote:

That's what the staging database is for in most extract scenarios, I find that even creating tempdb tables on the source systems are seldom OK.

Either way, by having an authorized user creating the table before SSIS runs, it's enough to give the SSIS context insert/delete rights.

Cheers/Kristian

When you say "staging database" are you referring to an object in the SSIS package or is that just a general term?

Basically, what I want to do is run through the entire input file, validate it, and record any errors I find before inserting any DB records.|||

In my use of temp tables and the OLE DB Source or Destination I have resorted to using the Advanced Editor where you can create the columns and mappings "by hand". An easy way to see how to set them up is to create a table in your dev db that looks like your temp table. Set up the source or destination using that table and map your columns. Then open up the Advanced Editor.

One thing I found with temp tables. Of course you know about the RetainSameConnection setting. I was using the same connection to create the temp table, fill it, then open it so that I could write to another table on the same connection. Where I write to another table on the same connection I had to create a new separate connection and use it to write to the other table. Otherwise it seemed the viability of the temp table was uncertain when I run a second parallel data flow task where that temp table would be used as well.

Put another way, I use one connection to create the temp table, fill it and read from it. I do not use that same connection to write or read from any other tables. This may not be effecient but it was the only way I could get things to work in parallel.

Hope this helps,
David Martin

|||We are loading data from several source systems simultaneously. Each system generates a set of identical files using the Raw Data File destination. All extracts share common load procedures that pick up the generated Raw Data files through an expression variable. It's a pretty slick way around temp tables and seems to work pretty fast. Would be nicer if there was a Recordset source and we didn't have to create the file on the server but for now there isn't one. You may want to look at that approach as a way around temporary tables.|||

Martin is absolutely right. Raw files are akin to temp tables.

He is correct that there is no recordset source (and with good reason) but you can build one yourself as explined here: http://blogs.conchango.com/jamiethomson/archive/2006/01/04/2540.aspx

You can't append to a recordset destination unfortunately. If you think that would be nice functionality to have then vote for it here: http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedbackid=6831c0b9-ef74-4132-a137-b073a950d3fd

-Jamie

|||I meant "staging database" as a general term, i.e. it's often useful for the SSIS part to have access to a database where it can write and read arbitrary tables, even if that database is different from both your source database and your target database.|||

<< we didn't have to create the file on the server >>

That's fine if:

a: Your package runs local to your source data (very rare I expect)

or

b: You do not need to reference temp data structures in the queries that populate the raw data files.

But when you need the functionality of temporary data structures when working with queries that reference remote source data, it's either heterogenerous joins (death to performance) or table variables.

Ken