[NIFI-11927] NiFi Registry fails to push flow to GIT using SSH - ASF JIRA

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 1.22.0, 1.23.0
Fix Version/s: 1.24.0, 1.23.1
Component/s: NiFi Registry
Labels:
None
Environment:
Java 17 & 11, CentOS 9, kernel 5.14.0-44.el9.x86_64

Description

I have been successfully running NiFi Registry version 1.20.0 (and others before it) for about a year now. Now I needed to switch server that Registry runs on, since the old one was outdated, and decided to set up NiFi Registry version 1.22.0.

The setup worked great and NiFi seems to have been running properly. However I recently noticed that it wasn't pushing anything to GIT. The changes are made, new versions are committed, they show up both in Registry GUI as well as the NiFi instance, but when I try to fetch some version it will not allow me giving me the following error:

I started investigating the issue and noticed that the GIT repository that NiFi Registry was using was several versions ahead of origin, no version got pushed to the GIT repo.

I investigated the logs and noticed following messages show up when I make new version in NiFi and commit it to registry:

2023-08-10 07:52:03,317 WARN [NiFi Registry Web Server-18] javax.persistence.spi javax.persistence.spi::No valid providers found.
2023-08-10 07:52:05,120 INFO [GitFlowMetaData Push thread] o.a.s.c.u.s.e.EdDSASecurityProviderRegistrar getOrCreateProvider(EdDSA) created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider
2023-08-10 07:52:05,326 INFO [GitFlowMetaData Push thread] o.a.s.c.i.DefaultIoServiceFactoryFactory No detected/configured IoServiceFactoryFactory; using Nio2ServiceFactoryFactory
2023-08-10 07:52:05,351 ERROR [GitFlowMetaData Push thread] o.a.n.r.p.flow.git.GitFlowMetaData Failed to push commits to origin due to org.eclipse.jgit.api.errors.TransportException: git@<OMITTED_FOR_PRIVACY>nifi/nifi-flows.git: remote hung up unexpectedly
org.eclipse.jgit.api.errors.TransportException: git@<OMITTED_FOR_PRIVACY>nifi/nifi-flows.git: remote hung up unexpectedly
        at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:147)
        at org.apache.nifi.registry.provider.flow.git.GitFlowMetaData.lambda$startPushThread$1(GitFlowMetaData.java:299)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.eclipse.jgit.errors.TransportException: git@<OMITTED_FOR_PRIVACY>nifi/nifi-flows.git: remote hung up unexpectedly
        at org.eclipse.jgit.transport.TransportGitSsh$SshPushConnection.<init>(TransportGitSsh.java:382)
        at org.eclipse.jgit.transport.TransportGitSsh.openPush(TransportGitSsh.java:159)
        at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:127)
        at org.eclipse.jgit.transport.Transport.push(Transport.java:1384)
        at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:137)
        ... 7 common frames omitted
Caused by: java.lang.NoSuchMethodError: 'java.lang.Object org.apache.sshd.client.future.ConnectFuture.verify()'
        at org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:189)
        at org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:142)
        at org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:99)
        at org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:235)
        at org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:1)
        at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)
        at org.eclipse.jgit.transport.TransportGitSsh$SshPushConnection.<init>(TransportGitSsh.java:358)
        ... 11 common frames omitted

I think the line Caused by: java.lang.NoSuchMethodError: 'java.lang.Object org.apache.sshd.client.future.ConnectFuture.verify()' is what's causing the issue with pushing to GIT. This does not show up on the old Registry instance when I push to that.

I checked the setup and compared it to the previous version running on old server. They have exactly the same configuration, save for new one running on Java 17 not 11 (although I tried running it on 11 too).

The SSH keys are valid and recognized by our Gitlab instance, I can push manually as the nifi user, there are literally no differences between the configuration on one server compared to another. I even checked the configuration inside .git folder in the flow_storage directory, it was the same as on the other server. The ~/.ssh folder for nifi user has similar data between two servers, both have keys and known hosts set up, with right permissions, etc.

I decided to try running version 1.20.0 of registry on new server, and it seems to have worked immediately, without any issue, it manages to push to GIT on its own, with no changes to config. I tested version 1.23 as well and it had the same issue, I haven't tested 1.21.0 though.

For more information, here is the structure I have:

├── nifi-registry-1.20.0
│   ├── bin
│   ├── conf
│   ├── docs
│   ├── ext
│   ├── lib
│   ├── logs
│   ├── run
│   └── work
├── nifi-registry-1.23.0
│   ├── bin
│   ├── conf
│   ├── docs
│   ├── ext
│   ├── lib
│   ├── logs
│   ├── run
│   └── work
└── nifi-registry-files
    ├── authorization-files
    │   ├── authorizations.xml
    │   ├── authorizers.xml
    │   ├── login-identity-providers.xml
    │   └── users.xml
    ├── certificate-files
    │   ├── keystore.jks
    │   └── truststore.jks
    ├── configuration-files
    │   ├── providers.xml
    │   └── registry-aliases.xml
    ├── database-drivers
    │   └── mariadb-java-client-2.7.4.jar
    ├── extension-bundles
    └── flow-storage
        └── Buckets...

Contents of the providers.xml:

<providers>    
  <flowPersistenceProvider>     
    <class&amp;amp;gt;org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class&amp;amp;gt;
    <property name="Flow Storage Directory">/disk1/nifi-registry/nifi-registry-files/flow-storage</property>
    <property name="Remote To Push">origin</property>
    <property name="Remote Access User"></property>
    <property name="Remote Access Password"></property>
    <property name="Remote Clone Repository"></property>
  </flowPersistenceProvider>    
  <extensionBundlePersistenceProvider>
    <class&amp;amp;gt;org.apache.nifi.registry.provider.extension.FileSystemBundlePersistenceProvider</class&amp;amp;gt;
    <property name="Extension Bundle Storage Directory">/disk1/nifi-registry/nifi-registry-files/extension-bundles</property>
  </extensionBundlePersistenceProvider>
</providers>

Contents of the .git/config file in the flow_storage directory:

[core]
  repositoryformatversion = 0
  filemode = true
  bare = false
  logallrefupdates = true
[http]
  sslVerify = false
  proxy = http://<PROXY_URL>:3128
[https]
  sslVerify = false
  proxy = http://<PROXY_URL>:3128
[user]
  name = nifi_user
  email = OMITTED_FOR_PRIVACY
[remote "origin"]
  url = git@<OMITTED_FOR_PRIVACY>nifi/nifi-flows.git
  fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
  remote = origin
  merge = refs/heads/master

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

image-2023-08-10-08-53-15-944.png
10/Aug/23 06:53
22 kB
Daniel Salwerowicz

Issue Links

is caused by

NIFI-11571 Upgrade SSHD to 2.10

Resolved

links to

GitHub Pull Request #7606

NiFi Registry fails to push flow to GIT using SSH

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Time Tracking