Owncloud + Piwik + Apache Access Log – Monitored Self-Hosted Enterprise File Sharing

If you aren’t familiar with Owncloud, it is a very cool open-source software package that runs on Linux Apache (or Nginx) that provides “dropbox like” functionality that you can host yourself.

This is a big deal for the tech-savvy average-Joe that is worried about keeping private data private (i.e. he doesn’t want all of his personal documents stored by Microsoft, or Dropbox, or Google, etc.) but still wants the “cloud-like” functionality of being able to securely access and sync files across multiple devices.

It is also a big deal for any enterprise that wants to use “cloud storage” but has to worry about all of the above due to data security requirements. It is self-hosted, so you know exactly where all of the data is and you have control over the security components protecting it. Citrix, Dropbox, and others have realized a growing need for this and have “enterprise” products that are in the same vein. They just cost a good bit of money, don’t always meet all of the stringent security requirements imposed on some types of data, and tend to be complex/cumbersome systems.

Owncloud also has an enterprise version of their software offering which runs upwards of $10k/year. When I did a comparison of the “enterprise” vs. “open-source” the only value I could see in going enterprise was support, and one additional module that does granular file-activity-logging (i.e. user jdoe, shared this file, on this date). Obviously support is support, you aren’t going to get enterprise support without paying an enterprise price. Writing that off, that just leaves the enhanced logging.

I don’t have the requisite skill-set to build my own logging module. But Owncloud is ultimately just a web application running on Apache, so why not track it like we would any other web application? Namely, using a site analytics tool and the Apache access log.

Enter Piwik… Piwik is like a “self-hosted” Google Analytics that is a more of a pain to setup and doesn’t have as many bells and whistles. But it is self-hosted, so your Analytics data is private. It is also open-source, so it goes with our ongoing theme of “free”.

This is by-no-means going to be a comprehensive setup guide for Piwik and Owncloud. Rather I am going to put some answers out there I came up with that as far as I could tell weren’t dealt with by anyone else.

Once you get your server initially configured, install owncloud in a webfolder and setup a virtual-host file for it. For example, here is a common place you might deploy it:

Now, when you go to install Piwik, if it is solely being installed to track Owncloud, then you should place it in a sub-folder off of your owncloud www directory. So you could unzip the Piwik application to somewhere like this:

This is going to make your life A LOT easier because you aren’t going to have to worry quite as much about something called the “Content Security Policy” that the Owncloud application takes advantage of. This is especially important for OwnCloud version 8.0.1+ because in the olden days there was an easy way to bypass the CSP, in newer versions there is not…

Okay, after you initially install Piwik, it is going to hand you some Javascript and tell you to place it everywhere you want to monitor stuff in your web-application. Just ignore all of that for now and let me tell you why… it has to do with new security protocols that won’t allow in-line java-script in a php file to be executed on the spot. So you will place your tracking code, but most (all?) browsers will no longer execute any of the script. So no tracking. To get around this, you have to put all of your script into a .js file and load that file, rather than loading the script. Here is how I did this for me…

In your piwik root directory, create file called “piwik-tracker.html” with the following contents:

In your piwik root directory, create a second file called “tracker.js” with the following contents:

Note, you need to update “var u” to reflect your site info.

Note, “var useridname” – that is the piece of code that was lacking from every other owncloud/piwik integration discussion. Other users used inline php to pull the username, but when the tracking code is moved into a separate js file, rather than being inline in an existing PHP document, the system no longer parses the included php. After some digging and testing (as I know very little about coding Java…) I came up with the native Javascript I could use to pull the element from the HTML that contained the logged-in user’s name. If you don’t get this working, your Piwik logs will not have any usernames and you miss out on a lot as a result.

Okay, so you created those two files. Now, we need to actually reference them in the appropriate places in our owncloud installation for tracking to actually work.

You need to add the following line:

in-between the

tags in the following files:

Once you do that, refresh your owncloud page a few times and then check Piwik. Data should be flowing in, and it should show the name of the logged-in user generating it. Sweet!

— Apache Access Log —

One of the frustrating items I couldn’t figure out was getting Piwik to track file shares and downloads. Owncloud uses Ajax (forgive me if I make a hash of all this as I am not a coder) quite heavily. It uses Ajax to generate contextual download links for files on the fly. I.e. you hover your mouse over “download file” and the associated link is not

If it were, I could figure out how to track that stuff in a heartbeat. Rather when you hover over a download link in owncloud, they ALL look like this:

You get the directory name, but not the file name. If someone has an answer for how to modify the piwik tracking code to catch the file-name, I am all ears! However I spent many long hours on it and never cracked it. So…

I turned to the Apache Access Log… First, in your virtual host file you probably need to clean-up the logging a bit. Because we are hosting Piwik in a sub-directory, your Apache access log file is going to contain logs for both Piwik and for Owncloud. If you want to turn off logging of everything that happens in Piwik (i.e. everything a user logged into Piwik does), your Virtual host file should be configured for logging with something like this:

This basically tells Apache a couple of things. First, I want my own error and access log that pertains to just this virtual host. Second, I don’t want to monitor any web-traffic in the “analytics” subfolder in my access log.

With that now done, your Access Log should be significantly less busy. If you are familiar with logging and log monitoring at all, you realize that getting rid of the “noise” in logs is like 90% of the game.

Okay, you can now “do stuff” on your owncloud site, and then tail your access log and you can see that Apache catches pretty much everything.

This is where I got lazy myself. Rather than write some nice scripts that will run and create reports regularly (I started on this path at first and will share what I came up with but this isn’t where I ended in all fairness), I just shipped our access log off to a SEIM server that parses and analyzes well enough for me. If you want to take this approach (which I kind of like because it also means there is an “off server” backup copy of all of my logs), there are some open-source SEIM options you could look into like Graylog2, AlienVault, and Syslog-NG to name a few… It is WAY out of the scope of this article for me to get into that…

If you want to keep it simple, and are okay with your logs being kept on server and just want an easy way to dig through them from time to time, I suggest “grep” and “awk”. Here are some one-liners I wrote to quickly get me some info. (writing them wasn’t quick as I had never used AWK before… however it is kind of amazing…)

Those are based on people interacting with the web application. I had just started working on mobile-client stuff. Ultimately I wanted to take it all, throw it into a script that could be run with some date/time parameters that would spit out a report (either a txt document or a CSV). Like I said though, got lazy and just sent it to SEIM :).