How to search all the deployment logs at once?
isme
Posts: 119
Hey RG,
It's easy to search the log for a single deployment.
You can use your browser's built-in search functions once you load the log in your browser.
But can you do the same across all logs for a given project?
I'm trying to find old logs that contained a particular error message to help explain why we changed one of our deployment procedures.
I don't see any way to do it in the web interface.
I can't find the log files on the file system, or in the RavenDB database.
Maybe I'm looking in the wrong place.
Do you know a way to do this?
Happy Christmas to you all!
It's easy to search the log for a single deployment.
You can use your browser's built-in search functions once you load the log in your browser.
But can you do the same across all logs for a given project?
I'm trying to find old logs that contained a particular error message to help explain why we changed one of our deployment procedures.
I don't see any way to do it in the web interface.
I can't find the log files on the file system, or in the RavenDB database.
Maybe I'm looking in the wrong place.
Do you know a way to do this?
Happy Christmas to you all!
Iain Elder, Skyscanner
Comments
C:\ProgramData\Red Gate\DeploymentAgent\Applications\.Agent\Logs
What I can't quite find is where the GUID these files use as their names is stored... the 'Deployments' document in the RavenDB doesn't seem to obviously hold it.
I'll see if I can find that out next week.
Redgate Software
What about SQL Server deployment logs?
SQL Server deployment agents run on the RGDM host.
On my RGDM host the folder
C:\ProgramData\Red Gatedoes not contain a folder called DeploymentAgent.
It contains just one folder called Licenses, which is empty.
For both Agent and SQL deployments, the logs are stored in the Raven DB as document attachments, however the Raven Studio viewer (http://localhost:10300) doesn't appear to offer any easy way to view these. You may be able to use the commandline tool to export them out though: http://ravendb.net/docs/server/administration/export-import
Redgate Software
Here's the Smuggler command I used:
It finished with this message:
It created a file called dump.raven approximately 13MB in size.
I opened it in Notepad++.
The attachments appear to be serialized as JSON, but they don't look familiar to me.
I can't paste a complete example because it contains binary data.
Each attachment looks a bit like this (I've elided the data stream):
{ "Data": "H4sIAAAAAAAEAOy9B2AcSZYl[...]ubBEevVOAAA=", "Metadata": { "ContentType": "text/xml" }, "Key": "attachments/tasks-34/output-log" }How should I decode this?
As an aside, apparently you can't view attachments in Raven Studio because the developers "don't want to encourage their use." See issue RavenDB-909 on the Hibernating Rhinos issue tracker.
The Data elements in those JSON objects look like they're base64 encoded. When we write the logs to the database, they're an XML string, so you should be able to get the XML back by reversing that. In C#, the code would be something like:
Redgate Software
The value of the Data property is not pure base64. There is some binary stuff in there as well.
In Notepad++ it looks like this:
If you ignore the the sequences of "NUL NUL EOT ÿ û" and copy the value to a new buffer, you get this:
When I decode that, I don't see any XML. All I get is a stream of bytes, mostly unprintable.
I get similar behavior when I use Raven's HTTP interface to fetch the same attachment.
This PowerShell script fetches the same attachment and prints the raw response:
The response looks like this:
The content type says text/xml, but that sure doesn't look like XML!
Any ideas on how I should further decode this output?
I'll dig a little more at this end...
Redgate Software
Redgate Software
Like you, my unzipped file was full of this:
{ "Data": "H4sIAAAAAAAEAIVSy27bMBC8B8g/LHio2wMp0bKetWzUcRoUqFujbtAedGEoWhEqkYJIWfbfl45sN0WD5rRY7HA4s7PT+b6uYCdaXSqZIkpcBEJylZeySFFntjhC89n11ZRxU+5KcwCLlzrZ6zxFj8Y0ieP0fU96j6i2cMauS52fq88b/ihqhi7g8nUwLqU2THKBQLJapGipelkplkPX5MwIsNoIDQi1Ci3QdDpFm45zoTWC0qpxkdUJMK1UMRu71MMuxZ4L1Ev8MKET+PTl41cAONNag2fm0VI0lTrUQhpYMckK0Z5+G8G2VTWMnolvRY4L+4pwVTsDAVfStKrSTn7iJkw3+7kwPMV07IU09L3If8Pq5n3RWakPNKAR8wMcbPMYT6I4wCyPQ+xRL4wiP/b88YAe6FN/Ejy1vCqtxJSOrq9eMri8Xdzf/W2QhpR6bhTBw8EIDUbBTZL9KKUVqrPvt6t1Zupm8SEeE1tfYJ1c1rZhuyPh2aG4xDJwrltVtKxeMsOyb3Z4Z0fZn7Wetpo9zdeM/7LdDbO5Z2Sg0dm/GeBL4qTW5X/V3b8e5FsawGrxDvRwM9uuqg7P7BB7487xdI71fOyz31lQzmgfAwAA", "Metadata": { "ContentType": "text/xml", "Content-Type": "application/json; charset=utf-8" }, "Key": "attachments/tasks-415/output-log" },I took the data value, and Base64 decoded it here, saving the output to a .bin file
I renamed the bin file to .7z and opened it up in 7zip, then extracted out the contents. I now have this:
Success!
So the steps seem to be:
- extract data
- unzip
- grab each data string
- base64 Decode
- unzip (again) the output
Hope that helps!
Redgate Software
I was about to reply and tell you that, having looked at the source code for our log serialization, we GZip the XML string before it is saved as an attachment. RavenDB Smuggler then base64 encodes that to store it as a JSON text field in its dump file.
If you go straight to the HTTP API to download the attachment (http://hostname:10300/static/attachments/tasks-{foo}/output-log), you'll get back a binary stream that can be GZip decompressed to get the XML log.
You can also get a list of all the available attachments by reading the JSON array at http://hostname:10300/static
Redgate Software
Redgate Software
I distilled your helpful advice into a simple PowerShell function to export the deployment logs.
I ran it on the RGDM host to dump XML versions of the deployment logs to a log directory.
PS C:\Users\admin_iain> Export-DeploymentLogs -TargetDirectory C:\Users\admin_iain\Desktop\Logs PS C:\Users\admin_iain> Get-ChildItem -Path C:\Users\admin_iain\Desktop\Logs Directory: C:\Users\admin_iain\Desktop\Logs Mode LastWriteTime Length Name ---- ------------- ------ ---- -a--- 14/01/2014 18:43 61741 deployments-7966.xml -a--- 14/01/2014 18:43 51797 deployments-7967.xml -a--- 14/01/2014 18:43 57730 deployments-7968.xml -a--- 14/01/2014 18:43 62936 deployments-7969.xml -a--- 14/01/2014 18:43 62936 deployments-7970.xml -a--- 14/01/2014 18:43 62936 deployments-7971.xml -a--- 14/01/2014 18:43 50796 deployments-7972.xml -a--- 14/01/2014 18:43 62936 deployments-7973.xml -a--- 14/01/2014 18:43 51797 deployments-7974.xml -a--- 14/01/2014 18:43 137176 deployments-7975.xml -a--- 14/01/2014 18:43 50909 deployments-7976.xml -a--- 14/01/2014 18:43 51797 deployments-7977.xml -a--- 14/01/2014 18:43 136534 deployments-7978.xml -a--- 14/01/2014 18:43 51797 deployments-7979.xml -a--- 14/01/2014 18:43 136534 deployments-7980.xml -a--- 14/01/2014 18:43 6305 deployments-8001.xmlNow I can search the files to find what I need.
In case anyone else wants to do this, here's the code I used.
function Export-DeploymentLogs { param ( [String] $TargetDirectory = '.' ) $AttachmentList = Invoke-RestMethod -Uri 'http://localhost:10300/static?start=0&pagesize=999999999' $AttachmentList | Where { $_.Key -like '*deployment*' } | Foreach { $AttachmentName = $_.Key.Split('/')[1] Invoke-WebRequest -Uri "http://localhost:10300/static/$($_.Key)" -OutFile "$TargetDirectory\$AttachmentName.xml.gz" gzip.exe --decompress "$TargetDirectory\$AttachmentName.xml.gz" } }It requires the gzip utlity to be on the path. I use version 1.2.4. You can download it from http://www.gzip.org/.
RavenDB pages the links to attachments through the static resource. Starting at 0 with the pagesize set larger than the number of attachments in the database, the static endpoint returns a link to all the attachments in the database.
The key property contains the word 'deployment' if the attachment is a deployment log. The function filters on this to ignore other types of attachment.
The full key is a relative URL for the attachment. If you fetch it, you get a gzip-compressed XML file.
gzip.exe deletes the input file and writes the decompressed version with an .xml extension.
In the end, the target directory contains a bunch of XML files.
Redgate Software
I'll copy the rest of my posted code there too.
Thanks again!