Troubleshooting the self-hosted integration runtime - Azure Data Factory and Azure Synapse (2023)

APPLIES TO: Azure Data FactoryAzure Synapse-Analyse

This article examines common troubleshooting methods for self-hosted integration runtime (IR) in Azure Data Factory and Synapse workspaces.

Collect self-hosted IR logs

For failed activities running on a self-hosted IR or a shared IR, the service supports viewing and uploading error logs. To get the Bug Report ID, follow the instructions here and enter the Bug Report ID to search for related known issues.

1. On the Service UI Monitor page, selectpipe runs.

2. Underactivities execution, nomistakeColumn, select the highlighted button to view the activity logs as below screenshot shown:

   • Azure Data Factory
   • blue synapse

   

   Activity logs are displayed for failed activity execution.

   

3. Select for more helpsend records.

   HimShare self-hosted integration runtime (IR) logs with Microsoftwindow opens.

   

4. Choose which records you want to send.

   • Forself-hosted infrared, you can upload logs related to the failed activity or all logs to the self-hosted IR node.
   • Forcommon infrared, you can only upload records related to the failed activity.

5. After the logs are uploaded, make a note of the report ID for later use in case you need further help solving the issue.

   

Generic self-hosted IR crash or error

Memory problem

• symptoms

  An OutOfMemoryException (OOM) error occurs when attempting to perform a search activity using a mapped IR or a self-hosted IR.

• cause

  A new activity can throw an OOM error if the IR engine is currently experiencing high memory usage. The issue could be caused by a high volume of concurrent activity and the bug is intentional.

• resolution

  Check resource usage and concurrent activities running on the IR node. Adjust the internal timing and trigger timing of activity runs to avoid running too many simultaneously on a single IR node.

Issue with concurrent jobs limit

• symptoms

  If you try to increase the UI concurrent worker limit, the process hangsUpdateCondition.

  Example scenario: The maximum value for concurrent jobs is currently set to 24 and you want to increase the number so that your jobs can run faster. The minimum value you can enter is 3 and the maximum is 32. You increase the value from 24 to 32 and then select theTo updateButton. The process hangsUpdateStatus as shown in the screenshot below. You refresh the page and the value is still displayed as 24. It wasn't updated to 32 as expected.

  

• cause

  The limit on the number of concurrent jobs depends on the logical core and memory of the computer. Try setting the value to something like 24 and see the result.

  addendum

  • For more information about the number of logical cores and how to determine the number of logical cores for your computer, seeFour ways to find the number of cores in your CPU in Windows 10.
  • To learn how to calculate the mathematical logarithm, go toLogarithmic Calculator.

Issue with Self-Hosted IR High Availability SSL Certificate

• symptoms

  The self-hosted IR worker node reported the following error:

  "Failed to extract shared states from head node net.tcp://abc.cloud.corp.microsoft.com:8060/ExternalService.svc/. Activity ID: XXXXX The X.509 certificate CN=abc.cloud.corp. microsoft . com, OU=test, O=Chain building failed Microsoft The certificate used has a trust chain that cannot be verified Replace the certificate or change the certificate validation mode The function The revocation server could not verify the revocation because the revocation server was down."

• cause

  When dealing with cases related to an SSL/TLS handshake, you may encounter some problems related to certificate chain verification.

• resolution

  • Here's a quick and intuitive way to troubleshoot an X.509 certificate chain compilation error:

    1. Export the certificate to be verified. To do this, proceed as follows:

       a. Select on WindowsStart, start writingcertificatesand then selectManage computer certificates.

       B. Using File Explorer in the left pane, locate the certificate you want to verify, right click and select itAll tasks>Export.

       

    2. Copy the exported certificate to the client computer.

    3. On the client side, run the following command in a command prompt window. necessarily replace<Certificate Path>j<Output TXT file path>with the real roads.

       Certutil -verify -urlfetch <certificate path> > <output TXT file path>

       For example:

       Certutil -verify -urlfetch c:\users\test\desktop\servercert02.cer > c:\users\test\desktop\Certinfo.txt

    4. Check the output TXT file for errors. The error summary can be found at the end of the TXT file.

       For example:

       

       If there is no error at the end of the log file as shown in the screenshot below, assume the certificate chain was successfully built on the client machine.

       

  • If an AIA (Authority Information Access), CDP (CRL Distribution Point), or OCSP (Online Certificate Status Protocol) filename extension is configured in the certificate file, you can check it more intuitively:

    1. Get this information by checking the certificate details as shown in the screenshot below:

       

    2. Run the following command. necessarily replace<Certificate Path>with the actual certificate path.

       Certutil -URL <certificate path>

       The URL recovery tool will open.

    3. To verify certificates with the filename extensions AIA, CDP, and OCSP, selectRecall.

       

       You have successfully created the certificate chain when the status of the AIA certificate isverifiedand is the status of the CDP or OCSP certificateverified.

       If the attempt to get AIA or CDP fails, work with your network team to prepare the client computer to connect to the target URL. It is sufficient if the HTTP path or the LDAP (Lightweight Directory Access Protocol) path can be verified.

Self-hosted IR failed to load file or assembly

• symptoms

  You receive the following error message:

  "The file or assembly 'XXXXXXXXXXXXXXX, Version=4.0.2.0, Culture=neutral, PublicKeyToken=XXXXXXXXXX' or one of its dependencies could not be loaded. The system cannot find the file specified. Activity ID: 92693b45-b4bf-4fc8 - 89da- 2d3dc56f27c3"

  (Video) Self Hosted Integration Runtime Auto Upgrade is not Working in Azure Data Factory Tutorial 2021

  Here's a more specific error message:

  "The file or assembly 'System.ValueTuple, Version=4.0.2.0, Culture=neutral, PublicKeyToken=XXXXXXXXX' or one of its dependencies could not be loaded. The system can not find the stated file. Activity ID: 92693b45-b4bf -4fc8- 89da -2d3dc56f27c3"

• cause

  In the process monitor you can see the following output:

  addendum

  In Process Monitor you can define filters as shown in the screenshot below.

  The error message above states that the System.ValueTuple DLL is missing from the fileglobal mount cache(GAC), these are noodlesC:\Programme\Microsoft Integration Runtime\4.0\Gatewayfolder orC:\Programme\Microsoft Integration Runtime\4.0\SharedHeretic.

  Basically, the process first unloads the DLLANDfolder then from theDividedNoodles and finally of noodlesportabookbinder. So you can load the DLL from any path that makes sense.

  

• resolution

  You will find themSystem.ValueTuple.dllfile inC:\Programmarchiv\Microsoft Integration Runtime\4.0\Gateway\DataScanbookbinder. To fix the problem, copy theSystem.ValueTuple.dllfile for theC:\Programme\Microsoft Integration Runtime\4.0\GatewayHeretic.

  You can use the same method to fix other assembly issues or missing files.

• More information on this topic

  The reason why you see themSystem.ValueTuple.dllunder%windir%\Microsoft.NET\assemblyj%windir%\mountis that this is .NET behavior.

  In the following error you can clearly see that theSystem.ValueTupleassembly is missing. This problem occurs when the application tries toSystem.ValueTuple.dllMontage.

  "<LogProperties><ErrorInfo>[{"Code":0,"Message":"Type initializer for 'Npgsql.PoolManager' threw an exception.","EventType":0,"Category":5,"Data " : {},"MsgId":null,"ExceptionType":"System.TypeInitializationException","Source":"Npgsql","StackTrace":"","InnerEventInfos":[{"Code":0,"Message" : "Could not load file or assembly 'System.ValueTuple, Version=4.0.2.0, Culture=neutral, PublicKeyToken=XXXXXXXXX' or one of its dependencies. The system cannot find the file specified.","EventType":0,"Category":5,"Data":{},"MsgId":null,"ExceptionType":"System.IO.FileNotFoundException","Source " :"Npgsql","StackTrace":"","InnerEventInfos":[]}]}]</ErrorInfo></LogProperties>"

  For more information on the GAC, seeglobal mount cache.

  See Also

  Create a self-hosted integration runtime - Azure Data Factory and Azure SynapseIntroduction to Generating Self-Signed Certificates - .NETConfigure Self-Service Group Management - Azure Active Directory - Microsoft EnterEnable self-service password reset for Azure Active Directory - Microsoft Enter

Missing self-hosted integration runtime authentication key

• symptoms

  The self-hosted integration runtime suddenly disconnects without an authentication key and the event log shows the following error message:

  "Authentication key has not yet been assigned"

  

• cause

  • Remove the node's self-hosted IR or logical self-hosted IR in the Azure portal.
  • A clean uninstall was performed.

• resolution

  If none of the above causes occur, you can leave%programdata%\Microsoft\Data Transfer\DataManagementGatewayfolder to see if theThe definitionThe file was deleted. If it's removed, follow the instructions in the Netwrix articleFind out who deleted a file from your Windows file servers.

  

Self-hosted IR cannot be used to join two on-premises datastores

• symptoms

  After you create self-hosted IRs for the source and target datastores, you should connect the two IRs together to complete a copy activity. If the datastores are configured in different virtual networks or the datastores cannot understand the gateway engine, you will get one of the following errors:

  • "The source controller cannot be found in the target IR"
  • "Target IR cannot access source"

• cause

  Self-hosted IR is designed to be the central node of a copy activity, not a client agent that needs to be installed for each data store.

  In this case, you must create the linked service for each datastore with the same IR, and the IR must be able to access both datastores over the network. It doesn't matter if the IR is installed on the source datastore or the target datastore or on a third-party machine. If two linked services are created with different IRs but used in the same copy activity, the target IR will be used and you must install the drivers for both datastores on the target IR machine.

• resolution

  Install drivers for the source and target datastores on the target IR and ensure that it can access the source datastore.

  If traffic between two datastores cannot pass over the network (for example, if they are configured in two virtual networks), copying to an activity may not complete even with IR installed. If you cannot complete the copy in a single activity, you can create two copy activities with two IRs, each in a VENT:

  • Copy an IR from Datastore 1 to Azure Blob Storage
  • Copy another IR from Azure Blob Storage to Datastore 2.

  This solution can simulate the need to use IR to create a bridge connecting two separate data stores.

A credential sync issue results in the loss of HA credentials

• symptoms

  If the data source credential "XXXXXXXXX" is removed from the current integration runtime node with payload, you receive the following error message:

  "If you delete the shortcut service in the Azure portal or the task has the wrong payload, please create a new shortcut service with your credentials."

• cause

  Your self-hosted IR is created in two-node HA mode, but the nodes are not in a permissions sync state. This means that credentials stored on the dispatcher node are not synchronized with other worker nodes. If the dispatcher node fails over to the worker node and the credentials are only present on the previous dispatcher node, the task will fail when trying to access the credentials and you will get the above error.

• resolution

  The only way to avoid this problem is to ensure that both nodes are in a permissions sync state. If they don't sync, you'll need to re-enter the new dispatcher's credentials.

Certificate cannot be selected because private key is missing

• symptoms

  • You have imported a PFX file into the certificate store.

  • When selecting the certificate through the IR Configuration Manager UI, you received the following error message:

    "Error changing encryption mode for intranet communication. Certificate '< is probablevalid name>' may not have a private key that can exchange keys, or the process may not have access rights to the private key. See the inner exception for more details."

    

• cause

  • The user account has a low privilege level and cannot access the private key.
  • The certificate was generated as a signature, but not as a key exchange.

• resolution

  • To operate the user interface, use an account with appropriate permissions to access the private key.

  • Import the certificate by running the following command:

    certutil -importpfx FILENAME.pfx AT_KEYEXCHANGE

Issue with unsynchronized self-hosted integration runtime nodes

• symptoms

  The self-hosted integration runtime nodes try to sync credentials between the nodes but get stuck in the process and after some time get the following error message:

  "Integration Runtime node (self-hosted) is attempting to sync credentials between nodes. This may take a few minutes."

  Use

  If this error persists for more than 10 minutes, check connectivity to the Dispatcher node.

• cause

  This is because the worker nodes do not have access to the private keys. This can be confirmed in the self-hosted integration runtime logs below:

  [14]0460.3404::05/07/21-00:23:32.2107988 [System] A fatal error occurred while attempting to access the private key of the TLS server credential. The error code returned by the cryptographic module is 0x8009030D. The internal error status is 10001.

  You will have no problem with the sync process if you use service principal authentication in the linked service. However, when you change the authentication type for the account key, the sync problem starts. This is because the self-hosted integration runtime service runs under a service account (NT SERVICE\DIAHostService) and needs to be added to the private key permissions.

• resolution

  To resolve this issue, you must add the self-hosted integration runtime service account (NT SERVICE\DIAHostService) to the private key permissions. You can apply the following steps:

  1. Open the Microsoft Management Console (MMC) run command.

     

  2. In the MMC panel, do the following:

     1. Selectarchive.
     2. ChooseAdd or remove addonno suspense menu
     3. Selectcertificatesin the "Available Plugins" section.
     4. SelectAdd to.
     5. Select "Certificate Plugin" in the pop-up panelComputerkonto.
     6. SelectNext.
     7. Select "Select team" in the areaLocal Computer: The computer running this console.
     8. SelectThe end.
     9. SelectOKin the Add/Remove Plugins section.

  3. In the MMC panel, continue with the following steps:

     1. Select from the list of folders on the leftRoot Console -> Certificates (Local Computer) -> Personal -> Certificates.
     2. Right click on theMDM Beta for Microsoft Intune.
     3. SelectAll tasksin the drop-down list.
     4. Selectmanage private keys.
     5. SelectAdd tounder "Group or user names".
     6. SelectNT-SERVICE\DIAHostServiceto grant full access to this certificate, request and secure.
     7. Selectcheck namesand then selectOK.
     8. Select in the "Permissions" sectioninquiryand then selectOK.

UserErrorJreNotFound error message when running a copy activity in Azure

• symptoms

  When attempting to copy content to Microsoft Azure using a Java-based program or tool (e.g. copying files in ORC or Parquet format), you receive an error message similar to the following:

  ErrorCode=UserErrorJreNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Java Runtime Environment not found. I am onehttp://go.microsoft.com/fwlink/?LinkId=808605Download and install on your (self-hosted) integration runtime node machine. Note 64-bit Integration Runtime requires 64-bit JRE and 32-bit Integration Runtime requires 32-bit JRE. ': The selected module cannot be found. (Exception from HRESULT: 0x8007007E), Source = Microsoft.DataTransfer.Richfile.HiveOrcBridge

• cause

  (Video) Azure Data Factory Self-hosted Integration Runtime Tutorial | Connect to private on-premises network

  This issue occurs for one of the following reasons:

  • Java Runtime Environment (JRE) is not correctly installed on your integration runtime server.

  • Your integration runtime server does not have the required dependency for the JRE.

  By default, the integration runtime resolves the JRE path using registry entries. These entries should be automatically configured during JRE installation.

• resolution

  Follow the steps in this section carefully. Serious problems can occur if you modify the registry incorrectly. Before you change itRegistry backup for recoveryif problems arise.

  To resolve this issue, follow the steps below to check the JRE installation status:

  1. Ensure that the integration runtime (Diahost.exe) and the JRE are installed on the same platform. Check the following conditions:

     • The 64-bit JRE for the 64-bit ADF integration runtime needs to be installed in the folder:C:\Programme\Java\

       Use

       the folder does notC:\Programme (x86)\Java\

     • JRE 7 and JRE 8 are supported for this copy activity. JRE 6 and versions prior to JRE 6 have not been validated for this use.

  2. Check the registry for correct settings. To do this, follow these steps:

     1. noTo runmenu typeedit againand then press Enter.

     2. In the navigation pane, locate the following subkey:

        HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Runtime Environment.

        nodetailsPanel should have a Current Version entry showing the version of the JRE (e.g. 1.8).

        

     3. In the navigation pane, look for a subkey that exactly matches the version (e.g. 1.8) in the JRE folder. There should be one in the details paneJavaInitiationForbidden. The value of this entry is the JRE installation path.

        

  3. Locate the bin\server folder at the following path:

     C:\Programme\Java\jre1.8.0_74

     

  4. Check if this folder contains a jvm.dll file. Otherwise, browse to the file in thebin\clientHeretic.

     

  Use

  • If any of these settings are not described in these steps, use theJRE window installerto solve the problems.
  • If all of the settings in these steps are correct as described, you may be missing a VC++ runtime library on your system. You can fix this problem by installing the VC++ 2010 redistributable package.

Self-hosted IR setup

Integration runtime registration failed

• symptoms

  Occasionally you may want to run a self-hosted IR on a different account for one of the following reasons:

  • Company policy does not allow the service account.
  • Some authentication is required.

  After changing the service account in the services area, the integration runtime may stop working and you may receive the following error message:

  "The integration runtime node (self-hosted) encountered an error during registration. Cannot connect to the integration runtime host service (self-hosted)."

  

• cause

  Many features are granted only to the service account. If you change the service account to a different account, the permissions on all dependent resources remain unchanged.

• resolution

  Access the integration runtime event log to verify the error.

  

  See Also

  Generate a self-signed certificate with a custom root CA - Azure Application GatewayPower BI usage scenarios: Managed Self-Service BI - Power BIStore and manage values ​​in variables in Power Automate - Power AutomateWork with self-service password reset

  • If the error in the event log is "UnauthorizedAccessException", do the following:

    1. Check aDIAHostServiceLogin service account in the Windows service area.

       

    2. Make sure the Netbooker account has read/write permissions on the%programadatos%\Microsoft\DataTransfer\DataManagementGatewayHeretic.

       • If the service logon account has not been modified, it should have read/write permissions by default.

         

       • If you changed the service logon account, mitigate the problem as follows:

         a. Perform a clean uninstall of the current self-hosted IR.
         B. Install self-hosted IR bits.
         C. Change the service account by doing the following:

         yeah Browse to the self-hosted IR installation folder and then browse to the folderMicrosoft Integration Runtime\4.0\CompartidoHeretic.
         ii. Open an elevated Command Prompt window. Substitute<user>j<password>with your own username and password, and then run the following command:
         dmgcmd.exe -SwitchServiceAccount "<user>" "<password>"
         iii. If you want to switch to the LocalSystem account, make sure you're using the correct format for that account:dmgcmd.exe -SwitchServiceAccount "NT Authority\Sistema" ""
         AgainNotUse this format:dmgcmd.exe -SwitchServiceAccount "Sistema local" ""
         4. Since the local system has higher rights than the administrator, you can optionally change this directly under "Services".
         v. You can use a local user/domain user for the IR service login account.

         i.e. Register the integration runtime.

  • If the error is "The service 'Integration Runtime Service' (DIAHostService) could not be started. Please check if you have sufficient permissions to start system services", then do the following:

    1. Check aDIAHostServiceLogin service account in the Windows service area.

       

    2. Make sure the login service account is overSign in as a servicePermission to start Windows service:

       

• More information

  If none of the above resolution patterns apply to you, try collecting the following Windows event logs:

  • Application and Services Logs > Integration Runtime
  • Windows Registry > Application

I can't find the Register button to register a self-hosted IR

• symptoms

  If you register a self-hosted IR, theRegistrationThe button does not appear in the Configuration Manager dashboard.

  

• cause

  As of the release of Integration Runtime 3.0, theRegistrationThe button has been removed from existing integration runtime nodes to allow for a cleaner and more secure environment. If a node has been registered with one integration runtime, whether it is online or not, re-register it with another integration runtime by uninstalling the previous node and then installing and registering the node.

• resolution

  1. In Control Panel, uninstall the existing integration runtime.

     Important

     Select in the next processSim. Do not keep any data during the uninstall process.

     

  2. If you don't have the Integration Runtime installer MSI file, go toDownload-Centerto download the latest integration runtime.

  3. Install the MSI file and register the integration runtime.

Unable to register self-hosted IR due to localhost

• symptoms

  You cannot register the self-hosted IR on a new machine when using get_LoopbackIpOrName.

  Debugging:A runtime error has occurred. The type initializer for Microsoft.DataTransfer.DIAgentHost.DataSourceCache threw an exception. An unrecoverable error occurred during a database query.

  Exception Details:System.TypeInitializationException: The type initializer for 'Microsoft.DataTransfer.DIAgentHost.DataSourceCache' threw an exception. ---> System.Net.Sockets.SocketException: An unrecoverable error occurred during a database query at System.Net.Dns.GetAddrInfo(String name).

• cause

  The problem usually occurs when localhost is resolved.

  (Video) 15. Setting up Self Hosted Integration runtime in Azure Data Factory

• resolution

  Use localhost IP address 127.0.0.1 to host the file to fix the issue.

Self-hosted setup failed

• symptoms

  You cannot uninstall an existing IR, install a new IR, or upgrade an existing IR to a new IR.

• cause

  The installation of the integration runtime is based on the Windows Installer service. Installation problems can occur for the following reasons:

  • Insufficient available disk space.
  • Missing permissions.
  • The Windows NT service is blocked.
  • The CPU usage is too high.
  • The MSI file is hosted on a slow network location.
  • Some system files or registry were accidentally touched.

The IR service account could not get access to the certificate

• symptoms

  When you install a self-hosted IR through Microsoft Integration Runtime Configuration Manager, a certificate with a trusted Certificate Authority (CA) is generated. The certificate could not be applied to encrypt communication between two nodes and the following error message appears:

  "Error changing the encryption mode for intranet communication: The integration runtime service account could not access the certificate '<valid name>'. error code 103"

  

• cause

  The certificate uses the Key Storage Provider (KSP) store, which is not yet supported. Currently, the self-hosted IR only supports Cryptographic Service Provider (CSP) storage.

• resolution

  We recommend that you use CSP certificates in this case.

  solution 1

  To import the certificate, run the following command:

  Certutil.exe -CSP "CSP oder KSP" -ImportPFX FILENAME.pfx

  

  solution 2

  Run the following commands to convert the certificate:

  openssl pkcs12 -in .\xxxx.pfx -out .\xxxx_new.pem -password Password: <enter password>openssl pkcs12 -export -input .\xxxx_new.pem -output xxxx_new.pfx

  Before and after conversion:

  

  

Self-hosted integration runtime version 5.x

For the upgrade to version 5.x of the self-hosted integration runtime we need.NET Framework 4.7.2 runtimeor later. On the download page you will find download links for the latest 4.x version and the two latest 5.x versions.

For Azure Data Factory v2 and Azure Synapse clients:

• If auto-update is enabled and you have already updated your .NET Framework runtime to 4.7.2 or later, the self-hosted integration runtime will be automatically updated to the latest 5.x version.
• If auto-update is enabled and you have not updated the .NET Framework runtime to 4.7.2 or later, the self-hosted integration runtime will not be automatically updated to the latest 5.x version. The self-hosted integration runtime remains in the current version 4.x. You may see a .NET Framework runtime update notice in the portal and self-hosted integration time client.
• If automatic update is disabled and you have already updated the .NET Framework Runtime to 4.7.2 or later, you can manually download and install the latest 5.x version on your computer.
• If auto-update is disabled and you have not updated your .NET Framework runtime to 4.7.2 or later. If you attempt to manually install the 5.x self-hosted integration runtime and register the key, you must first update your version of the .NET Framework runtime.

For Azure Data Factory v1 customers:

• Self-hosted 5.X integration runtime is not supported in Azure Data Factory v1.
• The self-hosted integration runtime is automatically updated to the latest 4.x version. And the latest version of 4.x does not expire.
• If you try to manually install the 5.x self-hosted integration runtime and register the key, you will be notified that the 5.x self-hosted integration runtime is not compatible with Azure Data Factory v1.

Self-hosted IR connectivity issues

The self-hosted integration runtime cannot connect to the cloud service

• symptoms

  When trying to register the self-hosted integration runtime, Configuration Manager displays the following error message:

  "Integration Runtime node (self-hosted) encountered an error during registration."

  

• cause

  The self-hosted IR cannot connect to the service backend. This problem is usually caused by network settings in the firewall.

• resolution

  1. Make sure the integration runtime service is running. If yes, go to step 2.

     

  2. If no proxy is configured on the self-hosted IR, which is the default configuration, run the following PowerShell command on the computer where the self-hosted integration runtime is installed:

     (Nuevo-Objeto System.Net.WebClient).DownloadString("https://wu2.frontend.clouddatahub.net/")

     Use

     The service URL may vary depending on the location of the data factory or Synapse workspace instance. To find the service URL, use the Manage page of the user interface in your data factory or Azure Synapse instanceintegration runtimesand click on your self-hosted IR to edit it. Select thereweTab and clickShow Service URL.

     The expected response is as follows:

     

  3. If you don't get the response you expected, use one of the following methods, as appropriate:

     • If you get a "Could not resolve remote name" message, there is a problem with the Domain Name System (DNS). Contact your network team to resolve the issue.
     • If you get a message that the SSL/TLS certificate is not trusted,certified checkto determine if it is trusted on the computer, then install the public certificate using the certificate manager. This action should mitigate the problem.
     • Go towindow>Event Viewer (Logs)>Application and Service Logs>integration runtimeand look for errors caused by DNS, firewall rules, or corporate network settings. If you encounter such an error, force close the connection. Because every company has its own custom network settings, contact your network team to troubleshoot these issues.

  4. If Proxy is configured on the self-hosted integration runtime, ensure that your proxy server can access the service endpoint. For a command example, seePowerShell, web requests and proxies.

     $usuário = $env:nome do usuário$webproxy = (get-itemproperty 'HKCU:\Software\Microsoft\Windows\CurrentVersion\InternetSettings').ProxyServer$pwd = Read-Host "¿Contraseña?" -assecurestring$proxy = neues Objekt System.Net.WebProxy$proxy.Address = $webproxy$account = neues Objekt System.Net.NetworkCredential($user,[Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices .Marshal ]::SecureStringToBSTR($pwd)), "")$proxy.credentials = $cuenta$url = "https://wu2.frontend.clouddatahub.net/"$wc = neues Objekt system.net.WebClient $wc .proxy = $proxy$webpage = $wc.DownloadData($url)$string = [System.Text.Encoding]::ASCII.GetString($webpage)$string

  The expected response is as follows:

  

  Use

  Rendering Considerations:

  • Check whether the proxy server should be included in the list of safe recipients. If so, make surethese domainsYou are on the safe recipient list.
  • Make sure the proxy server trusts the SSL/TLS certificate "wu2.frontend.clouddatahub.net/".
  • If you are using Active Directory authentication on the proxy, change the service account to the user account that can access the proxy as "Integration Runtime Service".

Error Message: Self-Hosted/Self-Hosted Logical IR/Self-Hosted Integration Runtime Node is Idle/"Running (Restricted)".

• cause

  The self-hosted embedded runtime node can have a status ofInactive, as shown in the following screenshot:

  

  This behavior occurs when nodes cannot communicate with each other.

• resolution

  1. Log in to the virtual machine (VM) hosted on the node. UnderApplication and Service Logs>integration runtime, open Event Viewer and filter the error logs.

  2. Check an error log for the following error:

     System.ServiceModel.EndpointNotFoundException: Unable to connect to net.tcp://xxxxxxx.bwld.com:8060/ExternalService.svc/WorkerManager. The connection attempt lasted a time interval of 00:00:00.9940994. TCP error code 10061: The connection could not be established because the target machine actively refused it 10.2.4.10:8060. System.Net.Sockets.SocketException: A connection could not be established because the target machine actively refused it. 10.2.4.10:8060 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.Sockets.Socket.Connect(EndPoint remoteEP) at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan time has expired)

  3. If you see this error, run the following command in a command prompt window:

     See Also

     Troubleshoot self-service password reset writeback issues - Azure Active Directory - Microsoft Sign-in

     Telnet 10.2.4.10 8060

  4. If you receive the "Failed to open connection to host" command line error shown in the screenshot below, contact your IT department for assistance in resolving this issue. After a successful telnet connection, contact Microsoft Support if you continue to have problems with the runtime node status.

     

  5. Check if the error log contains the following entry:

     Error Log: Cannot connect to job manager: net.tcp://xxxxxx:8060/ExternalService.svc/ There are no DNS records for host azranlcir01r1. Exception details for unknown hosts: System.ServiceModel.EndpointNotFoundException: There are no DNS records for host xxxxx. ---> System.Net.Sockets.SocketException: No host known in System.Net.Dns.GetAddrInfo(String name) in System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6) in System.Net.Dns . GetHostEntry(String hostNameOrAddress) at System.ServiceModel.Channels.DnsCache.Resolve(Uri uri) --- End of inner exception stack trace --- Server stack trace: at System.ServiceModel.Channels.DnsCache .Resolve( Hooray)

  6. To resolve the issue, try one or both of the following methods:

     • Put all nodes in the same domain.
     • Add the IP-to-host mapping in all hosts files of the hosted virtual machine.

Connectivity issue between self-hosted IR and your data factory or Azure Synapse instance or self-hosted IR and data source or sink

Troubleshooting network connectivity issues requires knowing how to collect network traces, understanding how to use them, andAnalyze or stream Microsoft Network Monitor (Netmon)before applying Netmon Tools in real cases of self-hosted IR.

• symptoms

  Occasionally, you may need to troubleshoot connectivity issues between the self-hosted IR and your data factory or Azure Synapse instance, as shown in the screenshot below, or between the self-hosted IR and the data source or sink.

  

  In both cases, the following errors can occur:

  • "An incorrect copy with error: Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Cannot connect to SQL Server: 'IP address'"

  • "One or more errors have occurred. An error occurred while sending the request. The underlying connection was closed: An unexpected error occurred while receiving. The connection was forcibly closed by the remote host's activity ID."

• resolution

  If you encounter the above errors, fix them by following the instructions in this section.

  • Collect a Netmon trace for analysis:

    1. You can configure the filter to monitor a server restart for the client side. In the sample screenshot below, you can see that the server side is the Data Factory server.

       

    2. After receiving the reset packet, you can find the conversation by Transmission Control Protocol (TCP).

       (Video) How to create Self-hosted Integration runtime in Azure Data Factory and Azure Synapse analytics

       

    3. Get the conversation between the Data Factory client and server below by removing the filter.

       

  • An analysis of the Netmon trace you collected shows that the Total Time To Live (TTL) is 64.Basics of IP Time to Live (TTL) and Hop LimitArticle extracted from the list below, you can see that it is the Linux system that is resetting the package and causing the disconnect.

    The TTL and Hop Limit default settings vary by operating system, as indicated here:

    • Linux kernel 2.4 (ca. 2001): 255 for TCP, User Datagram Protocol (UDP) and Internet Control Message Protocol (ICMP)
    • Linux kernel 4.10 (2015): 64 for TCP, UDP and ICMP
    • Windows XP (2001): 128 for TCP, UDP and ICMP
    • Windows 10 (2015): 128 for TCP, UDP and ICMP
    • Windows Server 2008: 128 for TCP, UDP and ICMP
    • Windows Server 2019 (2018): 128 for TCP, UDP and ICMP
    • macOS (2001): 64 for TCP, UDP and ICMP

    

    In the example above, the TTL is shown as 61 instead of 64 because the network packet has to go through several hops when it arrives at the destination, e.g. B. routers or network devices. The number of routers or network devices is subtracted to get the final TTL.

    In this case you can see that a reboot can be sent from the Linux system with TTL 64.

  • Check the fourth hop of the self-hosted IR to confirm the origin of the reset device.

    Linux system network package A with TTL 64 -> B TTL 64 minus 1 = 63 -> C TTL 63 minus 1 = 62 -> TTL 62 minus 1 = 61 Self-hosted IR

  • In an ideal situation, the TTL hop count would be 128, meaning the Windows operating system is running your Data Factory instance. As shown in the example below,128 minus 107 = 21 jumps, which means that during the TCP handshake, 21 hops for the packet were sent from the Data Factory to the self-hosted IR 3.

    

    So you need to involve the networking team to see what the fourth hop of the self-hosted IR is. If it's the firewall, like on the Linux system, check the logs to see why this device is dropping the packet after the TCP-3 handshake.

    If you're not sure where to look, try getting the Netmon trace of the self-hosted IR and firewall during the turbulent period. This approach allows you to determine which device might have reset the packet and caused the disconnect. In this case, you also need to involve your network team to move forward.

Analyze the Netmon trace

If you try to telnet8.8.8.8 888With the Netmon trace collected, you should see the trace in the screenshots below:

The images above show that it was not possible to establish a TCP connection with the8.8.8.8server-side at the port888, then you see twoSynReTransmitadd-on packages there. why sourceAUTOHOST2I couldn't connect8.8.8.8it will keep trying to connect with the first packet.

Success scenarios are shown in the following examples:

• if you can telnet8.8.8.8 53No problem, there is a successful TCP 3 handshake and the session ends with a TCP 4 handshake.

  

  

• The TCP 3 handshake above produces the following workflow:

  

• The TCP 4 end session handshake is illustrated by the following workflows:

  

  

Email notification from Microsoft about updating your network settings

You may receive the following email notification recommending that you update your network settings to allow communication with new Azure Data Factory IP addresses before November 8, 2020:

Determine if this notice affects you

This notice applies to the following scenarios:

Scenario 1: Outbound communication from a self-hosted integration runtime running locally behind a corporate firewall

How to determine if you are affected:

• youryou are notaffected if you set firewall rules based on fully qualified domain names (FQDN) using the approach described inConfigure a firewall configuration and an IP address whitelist.

• yourthey areaffected if you explicitly enable whitelisting for outbound IP addresses on your corporate firewall.

  If you are impacted, please take the following steps: Notify your network infrastructure team by November 8, 2020 to update your network configuration to use the latest Data Factory IP addresses. To download the latest IP addresses, go toDiscover service tags using downloadable JSON files.

Scenario 2: Outbound communication from a self-hosted integration runtime running on an Azure virtual machine in a customer-managed Azure virtual network

How to determine if you are affected:

• Ensure that there are outbound network security group (NSG) rules on a private network that contains the self-hosted integration runtime. If there are no exit restrictions, it will not be affected.

• If you have outbound rule restrictions, make sure you use service tags. If you use service tags, this is not affected. You don't need to change or add anything as the new IP range is below the existing service tags.

  

• yourthey areAffected if you explicitly enable whitelisting for outbound IP addresses in NSG rule configuration in Azure Virtual Network.

  If you are impacted, please take the following actions: Notify your network infrastructure team by November 8, 2020 to update the NSG rules in your Azure virtual network configuration to use the latest Data Factory IP addresses. To download the latest IP addresses, go toDiscover service tags using downloadable JSON files.

Scenario 3 - SSIS integration runtime outbound communication in a customer-managed Azure virtual network

How to determine if you are affected:

• Make sure you have outbound NSG rules on a private network that contains the SQL Server Integration Services (SSIS) integration runtime. If there are no exit restrictions, it will not be affected.

• If you have outbound rule restrictions, make sure you use service tags. If you use service tags, this is not affected. You don't need to change or add anything as the new IP range is below the existing service tags.

• yourthey areAffected if you explicitly enable whitelisting for outbound IP addresses in NSG rule configuration in Azure Virtual Network.

  If you are impacted, please take the following actions: Notify your network infrastructure team by November 8, 2020 to update the NSG rules in your Azure virtual network configuration to use the latest Data Factory IP addresses. To download the latest IP addresses, go toDiscover service tags using downloadable JSON files.

Unable to establish trust for SSL/TLS secure channel

• symptoms

  Self-hosted IR failed to connect to Azure Data Factory or Azure Synapse service.

  When checking the self-hosted IR event log or the client notification logs in the CustomLogEvent table, you find the following error message:

  "Basic connection closed: SSL/TLS secure channel trust relationship could not be established. The remote certificate is invalid according to the validation process."

  The easiest way to check the service's server certificate is to open the service's URL in your browser. For example, open theCheck the server certificate linkon the computer where the self-hosted IR is installed and view the server certificate information.

  

  

• cause

  There are two possible reasons for this problem:

  • Reason 1: The root CA of the service server certificate on the computer where the self-hosted IR is installed is not trusted.
  • Reason 2: You are using a proxy in your environment, the service server certificate is being overridden by the proxy, and the machine on which the self-hosted IR is installed does not trust the overridden server certificate.

• resolution

  • Reason 1: Make sure that the service server's certificate and its certificate chain are trusted by the machine where the self-hosted IR is installed.
  • Reason 2: Trust the replaced root CA on the self-hosted IR machine or configure the proxy to not replace the service server certificate.

  For more information on trusting certificates in Windows, seeInstalling the trusted root certificate.

• Additional information
  We have released a new SSL certificate signed by DigiCert. Make sure DigiCert Global Root G2 is on the trusted root CA.

  

  If you are not on the Trusted Root Certification Authority,download here.

Next Steps

For more troubleshooting help, see the following resources:

• Data Factory-Blog
• Data Factory feature requests
• Azure-Videos
• Stack Overflow forum for Data Factory
• Information from Twitter on Data Factory
• Data Flow Map Performance Guide