Connect ProxyFS to existing OpenStack Swift/Keystone installation #322
Comments
|
Welcome to ProxyFS, jnamdar! Apologies about the lack of the CentOS box... indeed, I believe the path to it needs tweaking. Nice to see you were able to get your Swift Proxy "NoAuth" pipeline up and working. Keystone Auth for the Swift API pipeline should "just work"... but I must admit to never having tried it. The pfs_middleware that enables ProxyFS sits after the Auth step in the pipeline, so we should be just fine. But if you are asking about Keystone Auth being integrated on the SMB side, I don't have any experience with that alas. Is that the need for you? Your attempt to get SMB mounting working to your modified setup is very interesting. I was first thinking that one of the authorizing steps was missed so I tried a couple of things.
What I note has usually happened when getting the error you received I just re-verified. I sent a SIGKILL to the 'proxyfsd Anyway, I'm thinking the proxyfsd process has failed. It would be very interesting to see the log from that process (/var/log/proxyfsd/proxyfsd.log). Next, it looks like you got a "500 Internal Error" when later attempting a Swift API method targeting this Swift Account. This makes total sense. What's going on here is there is a Header in the Swift Account that indicates this is a so-called "BiModal" Account... meaning that ProxyFS manages its content. In your proxy-server.conf's [filter:pfs] section, there are instructions for which proxyfsd instance (IPAddr & Port#) to contact (note the above comments about TCP Port 12345). The pfs_middleware attempts to contact this proxyfsd_host (this can actually be a list btw) asking ProxyFS for the IPAddr of the proxyfsd instance actually serving the Volume. In this case, it should be told "hey, yes, it's me - you've come to the right place". Anyway, it tries really hard to contact the proxyfsd process indicated...and if it fails, you'll see this "500 Internal Error". So with that, I believe all signs point to a failure of your proxyfsd process. If you can post the log file I mentioned, perhaps we can get to the bottom of this quickly. Now just one more thing to check out. As you noted, you needed to configure what we call the "NoAuth" Swift Proxy instance adjacent to your proxyfsd process. It should be serving localhost:8090 I believe. If you log onto the "node" running proxyfsd, you should be able to talk to the underlying Swift Account. Just make sure this is possible. Do a HEAD on the "BiModal" Swift Account... you should see the BiModal Header I mentioned earlier indicating it is, in fact, BiModal. If you do a GET on it, you should see at least one Container likely named ".checkpoint". This is the Container that receives all of the file system's metadata for that Volume in the form of a "write log" consisting of LogSegment objects numbered in ascending order (starting with 0000000000000002 or so). There will be holes as the same 64-bit "Nonce" number sequence is used for all sorts of things that must be named uniquely (e.g. Inode#'s, Container names, etc...). If you don't see the BiModal Header on the Account... or the .checkpoint Container in the Account, then what it sounds like to me is the Volume needs to be formatted. If you look in the start_and_mount_pfs script, you should see a function named One note about formatting. ProxyFS requires a bit more guarantee than an Eventually Consistent storage system such as Swift provides. It cannot handle getting back "stale" information that an EC system may supply if, say, you did a 2nd PUT of an Object (a subsequent GET could return nothing, the first version, or the 2nd version). To avoid this, ProxyFS never PUTs the same Object (or Container) twice. That's why I mentioned the "Nonce" above. As such, the only way it could "know" that it had never used a given Nonce Value before is if the Account starts out completely empty. So just a heads up that you should either never reuse an Account... or empty if first (e.g. with Hope all of this discussion gives you things to look at to get your system running again. I don't know which of these are going to help identify the issue... but hopefully at least one of them will. And, again, welcome to ProxyFS! |
|
Hello, thank you @edmc-ss for the thorough answer. Here is the output of my This line seems relevant : Edit: I take that back, this error doesn't appear anymore (I must've messed with something in the configuration). I formatted the volume by calling I do have a Regards Edit2: From the error , I get that the function I also dug further on the error I get when executing I fixed that but now, when trying to get a list of the containers in the , which basically means "connection refused". I'm not sure where the connection is trying to be made though... |
|
I'm trying to understand how ProxyFS stores objects. Reading this page, I get that when I create an object using Filesystem access (via the Samba share for instance), ProxyFS uses exclusively the NoAuth pipeline. This pipeline includes the The other pipeline (the usual one) seems to be only used when trying to access objects the usual way, and the If I'm right about these pipelines' roles, where exactly would authentication play a part when writing/reading object via Filesystem Access? I don't see how adding Keystone authentication to the usual pipeline would help since it's not used by Filesystem Access. Do you think steps such as asking for a Keystone token, and adding it to every request's header would have to be directly implemented in the SMB VFS/jrpcclient/jrpcfs/swiftclient layers? Looking forward to understanding ProxyFS better 😄 |
|
Hello again jnamdar... and I must applaud all the excellent questions/topics you are raising. Before going further, I want to make sure you are aware of the Slace "group" we've got that might make interacting with other ProxyFS folks more responsive. The slack group is here and you can invite yourself here. First off I want to address the only "interesting" line I saw in the proxyfs.log file you included: What's going on here is that ProxyFS attempts to provide a FUSE mount point for the file system at the path your .conf file specified. As is the case with all mounts, the mount point must be a directory. Indeed, on many systems, the directory must be empty. If either the directory does not exist... or is not empty (on systems that require it to be so), you'll get this error message... It's not fatal (hence, the "level=warning"), but may not be what you desire. A little background on how ProxyFS presents the file system via SMB vs NFS would be helpful here. Both smbd and nfsd are able to present a local file system to the network via those two protocols. For SMB, however, you probably noticed the "vfs" library (indeed, there are two: vfs, which is Samba-specific, and jrpcclient, which is generic and used by vfs). Using vfs (& jrpcclient), Samba is able to communicate directly (via a couple of TCP sockets per client) with the proxyfsd process. So, in the case of SMB/Samba, we are actually not presenting a file system that is locally available. NFS is a different beast. While at one time, the ProxyFS plan was to leverage nfs-ganesha, a tool that does for NFS what Samba does for SMB. As it turns out, nfs-ganesha has an "FSAL" mechanism that enables one to "plug in" a file system just like Samba's "VFS" mechanism. Hence, there was this intention to code up an "fsal" library to plug into nfs-ganesha that would leverage that same jrpcclient library to communicate with proxyfsd. Alas, other priorities arose and the team never implemented an "fsal" library. As a result, in order to present a file system via NFS, ProxyFS needed to expose the file system "locally". It does so via FUSE. As you can imagine, This whole SMB and NFS protocol thing is quite outside the purview of ProxyFS... so ProxyFS didn't want to "insist" on a FUSE exposure of the file system. As such, the FUSE mount step is allowed to fail since it isn't required (particularly in the SMB exposure case). It's very convenient, though, so even SwiftStack's Controller always ensures this directory exists (and is empty) so that the FUSE mount actually works. In any event, the /etc/exports is populated with a line (or more) to expose the FUSE mount point via NFS. I'm speculating that your original attempt had skipped the step of creating the directory to which the FUSE mount point was attempted. Now, here's something I do when ever I successfully mount... I "stat" the root of the mount. What I should see is that the Inode Number of the "root directory" of my mount is "1". If it's not "1", then you've probably not successfully mounted... or NFS is just exporting the directory and ProxyFS isn't presenting it. In any event, it's a handy quick check to make sure things are active. I'm gonna close this response as you've posted a follow-up... so I'll respond to that one in a separate post. |
|
Hello again jnamdar, Your speculation about the use of the "NoAuth" Swift Proxy versus the "normal" Swift Proxy is totally correct! Indeed, your question about "who provides authorization" is very key. As you can imagine, the world of SMB and NFS (distinctly different between them as well) is entirely different than for Swift & S3 API (also distinctly different between them as luck would have it). To put it vaguely, it is sometimes the protocol server and sometimes the file system and sometimes the Swift or S3 API pipelines that provide authorization. Let's talk about each: SMB:
NFS:
Swift API:
S3 API:
As you can well imagine, the impedance mismatch between access control among these four protocols is tremendous. What we generally tell our customers is that they should apply per-protocol access control and not rely upon any reasonable "mapping" between them. In other words, don't expect a "chmod" to alter the access control for users coming in via the Swift or S3 API. Hope the above makes some sense. I'll stop this post here to respond to your remaining topics in a subsequent post. |
|
Keystone Auth would be an awesome addition... though I don't understand how it might apply to NFS. As mentioned previously, the authorization strategy of NFSv3 is to entirely trust the "client". So let me just "punt" on that one. As for SMB, the protocol is very rich as it turns out. With SMB, it's the Server (i.e. Samba in this case) that provides the necessary Auth. SMB supports something called SPNEGO... the term actually is an acronym of sorts for "Security Profile Negotiation". SPNEGO supports all kinds of "plug in" Auth mechanisms spanning from the very old Lan Manager ("LM") all the way up to Kerberos (as implemented by Active Directory btw). While this is way out of my skill set, it should be possible to provide an Auth plug-in for Keystone...and I'd be surprised if somebody hasn't done that already. To my understanding, Keystone is well within the capabilities of the SPNEGO Auth processing model. I would solicit help from the Samba folks on that one... |
|
Re the error you are getting from the "normal" Swift Proxy, I'm just curious what your "pfs" middleware filter section looks like. It should be something like this: Notice that the port# is that same "12345" that is used by Samba's vfs/jrpcclient library. It uses the very same JSON-RPC mechanism to communicate with the ProxyFS instance (proxyfsd). The "host" should be any one of the PrivateIPAddr's that any of your ProxyFS instances are listening on. Indeed, it can be a list :-). What happens is that the pfs_middleware determines (from the Account's Header check) that the Account is being managed by ProxyFS and asks proxyfs_host:proxyfs_port which ProxyFS instance is actually managing the Account/Volume/FileSystem at this point in time. Kind of like a redirect. What I suspect you may be seeing is that your "normal" Swift Proxy cannot communicate with the desired ProxyFS instance... not sure though. |
|
It's a bit unfortunate that your starting point is ProxyFS/saio (or the "runway" equivalent) as that setup is trivially straight forward. I'm suspecting that your cluster's topology is quite a bit more complicated than just "localhost" :-). Perhaps you could focus on the PrivateIPAddr discussion above and see about making connections. |
|
I agree, the saio environment seems to be especially useful for test/development purposes, but it's the only thing I found to install ProxyFS. Is there any way I could check out another installation process that does not focus on hosting everything on a single machine? Regards |
|
I've actually made a start on a branch called "ArmbianExample" that will ultimately network 3 little ODroid HC1's into a small Swift cluster... with ProxyFS running on it in an HA arrangement. Beyond that, I guess I'd be remiss if I didn't plug SwiftStack's actual product. You can try that out and see what it does as well... It certainly "wizards" a lot of the setup we've been talking about. Happy to help you come up to speed on that product :-). |
Hi,
Firstly, thanks for this application and for giving us the opportunity to use it.
To try it out, I deployed ProxyFS on a CentOS 7.4 VM using the
Vagrantfilein thesaiosubfolder. By the way, the referenced vagrant box in this file seems to be down, I used this box (config.vm.box = "CentosBox/Centos-7-v7.4-Minimal-CLI"with thevirtualboxprovider to continue.After the
vagrant_provision.shfinished running, I compiled the ProxyFS project usingmake: everything went well. I then used the scriptstart_and_mount_pfsto mount the NFS and SMB share.I can create folders/files in both shares without issues, and then view everything with the
swiftCLI:I've been looking for a way to use ProxyFS in my existing OpenStack Swift/Keystone installation:
rocky(installed with this link)rocky(installed using this link)So far, I have been able to deploy a CentOS 7.4 VM using the
Vagrantfilein thesaiosubfolder. I removed everything regarding the installation of Swift (including the creation of the userswift) since I already have one installed.I then fiddled with the ProxyFS configuration on this VM to point to my existing Swift Proxy server. I installed the
pfsandmetamiddlewares on the machine hosting my Swift Proxy server, added them to the pipeline.I also launched another instance of the Proxy server listening on port 8090 with the
/etc/swift/proxy-server/proxy-noauth.cond.d/20_settings.conffile:/usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server/proxy-noauth.cond.dFinally I used the script
start_and_mount_pfs, after removing the lines about starting Swift, to launch ProxyFS and mount the NFS and SMB network shares.The NFS share seems to work well (I can create folders and write files), but I'm getting an error trying to mount the SMB one. Relevant info: since I haven't created a
swiftuser, I replaced it with thevagrantuser that was already existing in the VM in thesmb.conffile, and usedsmbpasswd -a vagrant.Command line error:
What I find in
/var/log/samba/log.smbdafter addinglog level = 3 passdb:5 auth:5insmb.conf:It looks like the samba authentication went well, but the relevant error to me are the following lines:
I tried troubleshooting this, but no luck so far. Would anyone be able to help on this?
Here's my
df -Houtput if needed:I also tried to get containers and objects I created via the NFS share with the Object Storage API, but I got the following error on my Swift Proxy server:
Relevant lines in
/var/log/messagesregarding the error:On another subject, does ProxyFS support Keystone authentication, instead of the
tempauthused in the main pipeline?More broadly, has anyone tried to connect ProxyFS to an existing OpenStack Swift/Keystone installation?
Regards
The text was updated successfully, but these errors were encountered: