How to search all the deployment logs at once?

ismeisme Posts: 119
edited January 16, 2014 8:14AM in Deployment Manager
Hey RG,

It's easy to search the log for a single deployment.

You can use your browser's built-in search functions once you load the log in your browser.

But can you do the same across all logs for a given project?

I'm trying to find old logs that contained a particular error message to help explain why we changed one of our deployment procedures.

I don't see any way to do it in the web interface.

I can't find the log files on the file system, or in the RavenDB database.

Maybe I'm looking in the wrong place.

Do you know a way to do this?

Happy Christmas to you all!
Iain Elder, Skyscanner

Comments

  • Hi Iain- the logs live on the agents, in this folder:

    C:\ProgramData\Red Gate\DeploymentAgent\Applications\.Agent\Logs

    What I can't quite find is where the GUID these files use as their names is stored... the 'Deployments' document in the RavenDB doesn't seem to obviously hold it.

    I'll see if I can find that out next week.
    Systems Software Engineer

    Redgate Software

  • Thanks, James.

    What about SQL Server deployment logs?

    SQL Server deployment agents run on the RGDM host.

    On my RGDM host the folder

    C:\ProgramData\Red Gatedoes not contain a folder called DeploymentAgent.

    It contains just one folder called Licenses, which is empty.
    Iain Elder, Skyscanner
  • I've checked with the team, and the SQL logs never get as far as ending up on disk; they get streamed straight between the special SQL deployment agent and Deployment Manager itself.

    For both Agent and SQL deployments, the logs are stored in the Raven DB as document attachments, however the Raven Studio viewer (http://localhost:10300) doesn't appear to offer any easy way to view these. You may be able to use the commandline tool to export them out though: http://ravendb.net/docs/server/administration/export-import
    Systems Software Engineer

    Redgate Software

  • Thanks, James.

    Here's the Smuggler command I used:
    Raven.Smuggler out http://localhost:10300/ dump.raven --operate-on-types=Attachments
    

    It finished with this message:
    Done with reading attachments, total: 1764
    

    It created a file called dump.raven approximately 13MB in size.

    I opened it in Notepad++.

    The attachments appear to be serialized as JSON, but they don't look familiar to me.

    I can't paste a complete example because it contains binary data.

    Each attachment looks a bit like this (I've elided the data stream):
       {
          "Data": "H4sIAAAAAAAEAOy9B2AcSZYl[...]ubBEevVOAAA=",
          "Metadata": {
            "ContentType": "text/xml"
          },
          "Key": "attachments/tasks-34/output-log"
        }
    

    How should I decode this?

    As an aside, apparently you can't view attachments in Raven Studio because the developers "don't want to encourage their use." See issue RavenDB-909 on the Hibernating Rhinos issue tracker.
    Iain Elder, Skyscanner
  • Hi Iain,

    The Data elements in those JSON objects look like they're base64 encoded. When we write the logs to the database, they're an XML string, so you should be able to get the XML back by reversing that. In C#, the code would be something like:
    Text.Encoding.UTF8.GetString(Convert.FromBase64String("...base 64 string from data..."))
    
    Development Lead
    Redgate Software
  • I was able to decode something, but it still makes no sense.

    The value of the Data property is not pure base64. There is some binary stuff in there as well.

    In Notepad++ it looks like this:

    R7uweSF.png

    If you ignore the the sequences of "NUL NUL EOT ÿ û" and copy the value to a new buffer, you get this:
    H4sIAAAAAAAEAOy9B2AcSZYlJi9tynt/SvVK1+B0oQiAYBMk2JBAEOzBiM3mkuwdaUcjKasqgcplVmVdZhZAzO2dvPfee++999577733ujudTif33/8/XGZkAWz2zkrayZ4hgKrIHz9+fB8/Ih7/Hu8WZXqZ101RLT/7aHe881GaL6fVrFhefPbRuj3fPvjo9zj6jZPH2bQtLov2OqX2y+bRu2b22Ufztl09unv36upqfHVvXNUXd/d2dnbv/t5fPH89neeL7CPbuLi58XaxbNpsOc0/SpfZIv/so6f5qqyuF/myPamWbV2VZV4fKxYfpdS2XTefffR6PZ3mTfNRWhBCOx8Rqmn6uKwujvZ2du9t7+5s39tNd/ce7e4+2t9PP9nZebSzk569ePZlmqav2+wiT3cfpU+rq2VZZRhzusqmb+njJj2vq0X6Yv153qZNXoNCv3HSh/kgCnPvUfrVqgexrdI2qy8I4CKbzotlHoN4fycK8d6j9Az0KcuspZnqvbj36H4XFUe/dFotVmXe5r8rTeRdEIepZKdU6P2+5PiIRkM8MtPW28s1DW3bvLB5hm49R19kb4FGlpZF06bVeUBN0/VtJua4LN278+wyTyd5vrQg8plPmj5xDE0MjPT12+tmmi2XeT1+/RMnz1OSnPG9hzvj+3s+qc7zfPYofZNni5MbufY9qBKZoRsRElQ+VjFsCaVp0V4/Otjd3b2LD4/X7fxutlrd5Xm8e7l79xm98Zrmupjm4+ZyevfjGJ0tak9Pn3z1uUMtt6Rq0quCiD/JafRVTZ8Xy0fp549+31f5LPs8a/Pf96W0OyGhyG/TxbNiGYx8K2vbfLFq010wyP07MRifdmFU6+VtJvM2rGV5w4hZh5XuGl4Ssev8GZXC2yoQlcE1t/1mJLA7upd1vspq4EHdS0cWo9tQpz+O8Xh8G8WnbGH6jOqwPv2+CjDszSyNQWmXqiQ0zezyajGbX+WT3d298Sy/HDfupbKaZuWj3Z17ZKq+tvhakijznczzKau1q3nezvP6FnyYFg3JTUqtbeMp5CWtlin9QTr+vYfDc9DB9KCDaW/yNqFItP0GUXnYQeV1LlI/na+Xb0XUdw/e/8W9r/viva/74v7XffH+133x06/74oOv++LB4ItOrKMvPvy6L+7ufO03h3nnpjeHmeemN4e556Y3h9nnpjeH+ce9GX9zmIFuenOYg2568zYs1LEMG3RRIxr6fF2W12pByPP4EA31oSa948HfxjqndweM3Os2X8FT7xJA4ziPEOogrOoKfWw39OL2p/u72e7Ow/3tnYPJbHs/f3Bv+yDPPt2+v/Pppw92zu/t7+azr2vr7u925ur0XT5dt/D9COWb8b1NYMOjN+5Ax2/v0+o9ESAGyRCJWjeBmGJ372Zy3J4gl1m5JhYgzp+5CO2SXKxsUsZDwi6IJ/kF+QLu7Y3vGL+cW/cs+c+6Jd8cmDox7buEkTePGalq3a7WFFtRRqC6ah7dQtV0CeYFkj0hGm90/nugvcEIdsXyvKoXLONw2iY5IfnoNpDSbYp2hefAuY+Eq16/fvqT3/3i6be/e/qESH9LOGdN+un+9qRo0y9fK5w39ToWXG1+W9XGo/RZVja3fV1DxnRNeQIdyevf5/Wb0y9u+f7J81dGLhX3dB9TsnNv9+F4b+/+bcGs6xrzMSvqfEqB5/Wj9OTR7/tdih2JaX7f5pr0weLebQn65esuSukXxbSumuq8TRVm+uJN+ilxz4NPd3YtEWCq0t1b9iIMFHS0N6b/dsa7O7cBofZlg5B7zC1ZAWsaw6iiWJIKoBjdvf77Pj39Se9Phvb7bhaVrtry0Pt42HR/HA0Ou7BUDYt9LwQwjeLrw1USvqHBi2IBmlYrp1mdp9llVpT4MyrOXUg0n9+jFAcyHKfLy6KullASL0gevp9+ln5M5IzlUzaBUe1gQbBlel8gr/Iyz5r8xXoxyWsG45Hoa8Kq2rxhUO8L4KmX4l3gx9cDo/6gJUyHCb4muJ8UOfxAGnm4HS9nPsxh0XzvPurqp0nHufGD84W2N4EaAPU0b6Z1sWoNru8L5k3WvD2b8ast/dps39t/XxDKXLMn1wyGEpC/v0lavi8oIcWHgXKs6knz1xBDB+dl1jRXVS1U+pb3fA1YX5GtXZrpx/hm9qtbQ1P1/OFoKaAAJ1gP4PUTayiL57A59a0BElPO1lOh9uufeM4wOFv8nnDUKDs5+UVlF5Y2uTXIniJjG+6LubXgm2GqXaOlrksCQC+7tHm9XprcnzNNNEN5vdkQKUhC8xVBoDyuYDleNc4h2fze8WpF5nU8rZbnBeV/62zZwLm9VZBCr3+1mhFtUlpUeJ23iHiauwRqSfqFxve6RVKZs5rawVYTS93HAJ9Uq2vjsdxqpt6lT7M2m5A+sY6Ok5B0Va4RH2wtqxbIlsUUVp7T7XNy4fWN2yIHYr83pU+YAmvyMs7OXt/ynac5AmA7oPOi3EyMDj9UTXtLNJWKz6uKs9bEAvT2FQnLPCcOFUvRsKs/ew82U6gvKgthKk5cACKlfsD3tFxzK8fQRzJbXqfCWutaAjLLwvInSHYrZg6gOnamHrAwEWFpdG3k5uZelBSv4Q5wSixAGu8/ei9v/PelVMT+Mm8Vgw19/yz07X/VV7Dj/F3+c4KWAPh9Z/l5ti5/biijKFQU4dXFLG9+XyRS7F//r0Fp9/+NOO39vxGpT/9fiNT7EeqHg9P/6+hE+Y3/t6HEWdT/tyFVrShN1vy/jlaUl5z9v05LAan/1wkfkLq3AamfM6T2/9+I1P3/NyL1/zrlCaQe/L8RqYP/tyHVXOz+/v+v1FUGsb3/tyL2/zqd9f9GM3i2bPMLgfeG0Pt/ZTwRw/H/deIQRfL/dbY8iuUHWocfApb/L/Uhe3gCyf/3cyaw/H8/ZwLL/9epcE2KiQb6ucyQbcRs74eK2nuh9un/O1FDMH8zYns/fMSM5vt/JXLil/6/FrXbiMHPEWr3/t+L2v77ovbDQ+3+/3tRu41e+zlC7cH/e1E7+H8nai7m/n81ev8v0W9D6P2/VMe9rznV5ds3c7dMPs+adJLnS1qlbVpZcG6r98PvNsQ4O0+vq3V6Va3LWVoWb/O09ZBoK8IhwIBWjNOsbPN6SVS7zFMsB4B8o3RV5lmTp80qnxbn1wzmMquLbFLm6cev8tnnWZu/FLhPi5pWoqv6+mXWzj++DZq3WtPfvKDfJfYJDQkge+AepR+/D5XJ5US3v6/rvT+ivUf3P+1xIfr64K6IiRdE+Tafjd+j12rdrtbto9u88Rsnv3HyVYP5XuRp+DxKX79++pPf/eLpt797+mR3d+93+42T03f5dM3i9LIqi+m1bfnkepU1zW+cvKyrad40NI3H9XRetMQH6zqnBsdfPP10/zdOvlvVPNGWRQTA+zI+T2rDk/qT9AsQEjz2xjsYEf5TXKmzlWstLPBevfVnZTymyfhqqXKDHtJLxcJH8nVbrVb49vUvKn9iXbX5STad58+rbEYMmdeXxTSXRtv6Fyiuv308/E669fonng98d+fjtCGA6XlG2g4sc/yhY320t5NO51n9aI+k7hPLGwHav7BsD/3/p+nv1shXvz/Y6jdO8MYnJJEUntOUny3PK5/Lvlzly9O6rupHNLRrilUWY4Ws3GT+PKmWbV2Rpqof9T65k37Px+n7I0PKk2qxyJaz03fTfAXONdg8W5fl9U+ss7I4L/IZ9382wwwQFH31GVNx9EUxraumOm/HTpeMFSzh5trrZ9TDz9XkY1arFkodbLAipTlbs5qHuj4nQlVXwCkXap9Ia+aYYbCpzmVK7A1ttCbrkH48/vib5K6fRe46Kasm/38Pe53AEr+oWo9tvh6DnS2LtiDwDU0yuSIXNK+/cXJS5xmrvCmmkf4OXKBXebMuYRa+93o9xci/n76A2h+ee7/p01yUJyA5DYKXU3475ddTed+QxwegH93Y5RtKWBAJztPLe+Pd8e7Og/HOKB2/OH2TPqvp3SsyIenlPnTWzr3dh+O9vfui8OFerY1WTgv4V0QVuIJddX0zs//Gyat8UV2i8ellvmyfVxdpU63r6SbUqRsDZkMPjBgA1+iBZu+8rhYsoI1w5/h2cK7Ig2yEtudgtkF4fdrA+fQdi+OW2q2YcUhXmFZlBj/XmLbfOHnKRqHnWf/GyU+evnry5etTMvIBHPjNVzX5AOnH8sJd60mr0/xxelW08/Q3TnoN7saXPj72OvtSwTuEevAN+FtCv81IQq9/wwC0oXbVeW1gGGGrQew3gYZisHyerVbkpeks/TzSBa/WEgFmaVtnyyabEpsbtubhs0w8MeqBhUXJlq7mCHOqc/6w845H2tuoEK+5saEblMctxd4GjoHs2xhu7BmB22sueh/Mlx47hmF8RHGEpLGKI8BgRJHjjGkGI1W02rijhQVe0ILB9cDZtv0J7Gmv770++X56Ms+WF7mz25Cf11+dnJy+fn2bMOhVTnEK/JsZz30kaNy5DZjj9CqfGPFFxoG1NJys82q9nI1oJuiv9OzsdSc7cVVQV+S2rfKaYmCKdsfp70NxO8lVum4kaNcYm3yH4+/mk9fQRpQi+WkKo1wgXmi4ny1bDZcXxALpF9mSwvIaqqx5W6z6/W8OLjWifpoTxTmekjD//YK27ldj99V4uV697WdQIvS9VZ7gZdW0T20AcxuwLyoLYiqZkBAGzWoN/5mmkBnzMVTwkbh6afr4LvFncVm01/pR54PgT++P/ycAAP//ubBEevVOAAA=

    When I decode that, I don't see any XML. All I get is a stream of bytes, mostly unprintable.

    I get similar behavior when I use Raven's HTTP interface to fetch the same attachment.

    This PowerShell script fetches the same attachment and prints the raw response:
    $Response = Invoke-WebRequest 'http://localhost:10300/static/attachments/tasks-34/output-log'
    $Response.RawContent
    

    The response looks like this:
    HTTP/1.1 200 OK
    Transfer-Encoding: chunked
    Raven-Server-Build: 2666
    ContentType: text/xml
    Temp-Request-Time: 0
    Date: Mon, 13 Jan 2014 18:43:44 GMT
    Expires: Sat, 01 Jan 2000 00:00:00 GMT
    ETag: 02000000-0000-0001-0000-00000000002B
    Server: Microsoft-HTTPAPI/2.0
    
    .?........??.`.I?%&/m?{.J?J??t?.?`.$??@.???????.iG#)?*??eVe]f.@??????{????{????;?N'????\fd.l??J???!???.?~|.?".?.?.ez??MQ-??hw??Q?/???X^|???=?>???8??????-.??:??????f??G??]
    =?{???j|uo\?.w?vvv???_<.=????#?????v?l?l9??J??"????????.??=??m]?e^.+..???]7?}?z=??M?QZ.B;..?i???.??vv?m??l??Mw?.??>??O???y?????x?e?????"Ow.?O??eYe.s???o??&=??E?b?y??M^?B?
    q??? 
    s?Q???.???6?/.?"???e.?x.'
    ?????)?????????~..G?tZ-Ve???+M?].??d?T??????FC<2????5m???y?n=G_do?F??E???y@M??m&??,????2O'y?? ??O?>qM?????f?-?y=~?.'?S?????;??{>???|?(}?g??.??=?.??...T>V1l	?i?^?:?????
    .????n?Z??y?{?{?.?????????rz??.?-jOO?|??C-??j??????i?UM?.?G???~?W?,?<k?????;!??o???b.?|+k?|?j?]0??;1.?vaT??m&?6?ey??Y???.^.???.???*.??5??f$?;??u??j?A?KG.??P????x|.??la???
    ?>??
    0??,?Ai??$4???j1?_??????,?.7????f????{d????Z?(????)???y????.|?.?MJ?m?)?%??)?A:????s?????io?6?H??.Qy?A?u.R????oE?w.?????????????}???}????????????N??/>??/??|?7?y??7????7???
    ?7????7????..s??nzs??nz?6,??.tQ#.?|]??jA???.??&?????:?w.???6_?S?.@?8?.? ??
    }l7???????????????l{?.po? ?>??????.v???????k???v???]>]???.????M`??7?@?o???=. .?.?Z7??bw?fr?? ?Y?&. ???.??\?lR?C?.?'?.?.????.??[?,???%?.?:1????7?.?j???.[QF??j.?B?t	?.?=!.o
    t?{???.v????.,?p?&9!??6??m?v???????^?~????????{??H.K8gM?????h?/_+?7?:.\m~[????YV6?}]C?tMy..???????/n????WF..?t.S?so??xo??m???.?1+?|J?????????]?.?i~???????m	???.J?.??????M
    .f??M?)q??Owv-.`???[?".t?7??v??;?.??e??{?-Y.k.???X?
    ?.????>=?I?O???n.??????x?t...??T?}/.0????U.????b.?V+?Y???eV??3*?]H4???..2.??????P./H.??~?~L???S6?Q?`A?ez_ ??2????z1?k.???k????aP?????]???.???%L?	?&??.9?@.y?./g>?a?|?>???
    I???????7?..?4o?u?j???M??=???-??l??._.?\?'??.???IZ?/(!???r??I??.C.?e?4WU-T???|X_??]????f??[CS???h)?.'X.??.k(???9??..S??S????x?08[??p?(;9?Ee.?6?5??"c.??????a?]???K.@/??y
    ?^???3M4Cy??.)HB?.A?<?`9^5?!????jE?u<???.?.?l????U?B?.??.mRZTx???x??.jI?????ER?????V.K??.?T?k???j???O?6??>?????tU?..l-?.???.V???sr????".b?7?O?.k?2??^????9.`;????L?.?TM{K4
    ????????.??.	?<'..K????{.6S?/*a*N\."?~???\s+??G2[^??Z?Z.2???'Hv+f.?:v?.?0.aitm???^?.??.pJ,@.??z/o???T??2o.?}?,??.?W???]?s??.?}g?y?..n(?(T.???,o~_$R?_??Ai???8???.?O?_???.
    ???????D??????Y???!U?(M????V?????NK.???	.???.??3????????7"??:?	?.??.????!?\??????U.????"??:???F3x?l??????.e<.???u?.E??u?<??.Z?..??/?!{x.???s&??.?g.????pM??.????m?l????{?
    ???;QC0.3b{?|????.%r????.?????.j??????????C???{Q??^?9B???{Q;?.'j.??.5z?/?oC???T???9???7s?L>??t??KZ?mZYpn????6?8;O??uzU??YZ.o????h+?!??V???l?zIT??S,.?|?tU?Y???*?.???2??lR
    ???????Y??.?O??V????e??????Z?????%?		 {?.?.?.???D????????G???q!??????.D?6?????j????????.'?q?U??^?i?<J_?~????????{?dww?w????w?t????*???m??z?5?o????i?44???t^??.?:?.?_<?t?7N
    ?[?<??E.??2>Oj?????..<??;..?S\???k-,?^??ge<???j?r?.?K??G?u[?V???/*.b]??I6????lF????4?F??.(??}<?N???'?.|w???!??yF?.,s??c}???N?Y?h???.?..???l.???????W????7N??'$?.????-?+?
    ??\??????G4?k?U.c???d?<??m]???.?>??~????#C??j?????w?|.?5?<[???O???8/?.?.6?..}?.Sq?E1???:o?N??.,????g?????cV?.J.l?"?9[????>'BUW?).j?Hk??a???eJ?m?&??~<??????E?:)?&?..{??.?
    ?Z?m?.??-?? ?M2?".4??qrR?.??)???.\?Wy?.a.??z=???????.?{???\?' 9??S~;??Sy???.?.???.JX.	???{????????(.?8}?>???+2!??>t????????????^??VN?WD.??]u}3?????|Q]???e?l?W.iS???&??..
    fC.?..???f???.,??p??vp???l???`?Ax}????.??????qHW?Ve.?????8y?F??Y???O??z???S2?..??W5?.????]?I???qzU???7Nz???>>?:?R?;?z??[B??HB?.?.??v?ym`.a?A?7??b?|??V???,?<?.??..fi[g?&?.
    ?.????L<1???E????.s?s????G???.????.??-??.????.n?.??k.z.??.;?a|Dq????#?`D???i.#U?????.^????????	?i???>?~z2??.??????_????~}?0?UNq
    ??.?}$h?????*?.?E???4???j???h&?????u';qUPW?????b`?v???Cq;?U?n$h?.?|???????F?"?i
    ?\ ^h??-[?.?.?.????.??y[???o..5?~?.?9??0?????Wc??x?^??gP"??U??e??Om.s.?/*b*??..?j????.?1T???zi??.?gqY???Q???O???'..????Dz?N..
    

    The content type says text/xml, but that sure doesn't look like XML!

    Any ideas on how I should further decode this output?
    Iain Elder, Skyscanner
  • Hmm, running the command against my DB gives a file that's all binary with no json at all. Curious.

    I'll dig a little more at this end...
    Systems Software Engineer

    Redgate Software

  • OK, I needed to open the dump file inside dump.raven in 7Zip first. :)
    Systems Software Engineer

    Redgate Software

  • OK, some progress.
    Like you, my unzipped file was full of this:

    {
          "Data": "H4sIAAAAAAAEAIVSy27bMBC8B8g/LHio2wMp0bKetWzUcRoUqFujbtAedGEoWhEqkYJIWfbfl45sN0WD5rRY7HA4s7PT+b6uYCdaXSqZIkpcBEJylZeySFFntjhC89n11ZRxU+5KcwCLlzrZ6zxFj8Y0ieP0fU96j6i2cMauS52fq88b/ihqhi7g8nUwLqU2THKBQLJapGipelkplkPX5MwIsNoIDQi1Ci3QdDpFm45zoTWC0qpxkdUJMK1UMRu71MMuxZ4L1Ev8MKET+PTl41cAONNag2fm0VI0lTrUQhpYMckK0Z5+G8G2VTWMnolvRY4L+4pwVTsDAVfStKrSTn7iJkw3+7kwPMV07IU09L3If8Pq5n3RWakPNKAR8wMcbPMYT6I4wCyPQ+xRL4wiP/b88YAe6FN/Ejy1vCqtxJSOrq9eMri8Xdzf/W2QhpR6bhTBw8EIDUbBTZL9KKUVqrPvt6t1Zupm8SEeE1tfYJ1c1rZhuyPh2aG4xDJwrltVtKxeMsOyb3Z4Z0fZn7Wetpo9zdeM/7LdDbO5Z2Sg0dm/GeBL4qTW5X/V3b8e5FsawGrxDvRwM9uuqg7P7BB7487xdI71fOyz31lQzmgfAwAA",
          "Metadata": {
            "ContentType": "text/xml",
            "Content-Type": "application/json; charset=utf-8"
          },
          "Key": "attachments/tasks-415/output-log"
        },
    


    I took the data value, and Base64 decoded it here, saving the output to a .bin file

    I renamed the bin file to .7z and opened it up in 7zip, then extracted out the contents. I now have this:
    <?xml version="1.0" encoding="utf-8"?>
    <activity xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Download update 1.0.16.10" status="Success" id="0">
      <log>2013-01-30 13:57:14 INFO   Downloading update 'Deployment Manager 1.0.16' from 'http://www.red-gate.com/updatecontrols/download.aspx?etc=-1237175385&guid=b1618a56-6fd9-4896-ad97-313788593525&update=546&client=1'
    2013-01-30 13:57:14 DEBUG  Downloading 17113088 bytes to C:\Windows\TEMP\tmpBA92.tmp
    2013-01-30 13:57:44 INFO   Saving downloaded update to C:\ProgramData\Red Gate\DeploymentManager\Data\PackageCache\.updates\Deployment Manager-1.0.16.10.msi
    2013-01-30 13:57:44 INFO   Update 'Deployment Manager 1.0.16' (16 MB) successfully downloaded.
    </log>
    </activity>
    

    Success!

    So the steps seem to be:

    - extract data
    - unzip
    - grab each data string
    - base64 Decode
    - unzip (again) the output

    Hope that helps!
    Systems Software Engineer

    Redgate Software

  • Amazing sleuthing without using the source code James :)

    I was about to reply and tell you that, having looked at the source code for our log serialization, we GZip the XML string before it is saved as an attachment. RavenDB Smuggler then base64 encodes that to store it as a JSON text field in its dump file.

    If you go straight to the HTTP API to download the attachment (http://hostname:10300/static/attachments/tasks-{foo}/output-log), you'll get back a binary stream that can be GZip decompressed to get the XML log.

    You can also get a list of all the available attachments by reading the JSON array at http://hostname:10300/static
    Development Lead
    Redgate Software
  • Actually I did poke the source code and saw some references to gzip so figured it was worth a punt... ;)
    Systems Software Engineer

    Redgate Software

  • Thanks for your help!

    I distilled your helpful advice into a simple PowerShell function to export the deployment logs.

    I ran it on the RGDM host to dump XML versions of the deployment logs to a log directory.
    PS C:\Users\admin_iain> Export-DeploymentLogs -TargetDirectory C:\Users\admin_iain\Desktop\Logs
    
    PS C:\Users\admin_iain> Get-ChildItem -Path C:\Users\admin_iain\Desktop\Logs
    
    
        Directory: C:\Users\admin_iain\Desktop\Logs
    
    
    Mode                LastWriteTime     Length Name                                                                                                                                          
    ----                -------------     ------ ----                                                                                                                                          
    -a---        14/01/2014     18:43      61741 deployments-7966.xml                                                                                                                          
    -a---        14/01/2014     18:43      51797 deployments-7967.xml                                                                                                                          
    -a---        14/01/2014     18:43      57730 deployments-7968.xml                                                                                                                          
    -a---        14/01/2014     18:43      62936 deployments-7969.xml                                                                                                                          
    -a---        14/01/2014     18:43      62936 deployments-7970.xml                                                                                                                          
    -a---        14/01/2014     18:43      62936 deployments-7971.xml                                                                                                                          
    -a---        14/01/2014     18:43      50796 deployments-7972.xml                                                                                                                          
    -a---        14/01/2014     18:43      62936 deployments-7973.xml                                                                                                                          
    -a---        14/01/2014     18:43      51797 deployments-7974.xml                                                                                                                          
    -a---        14/01/2014     18:43     137176 deployments-7975.xml                                                                                                                          
    -a---        14/01/2014     18:43      50909 deployments-7976.xml                                                                                                                          
    -a---        14/01/2014     18:43      51797 deployments-7977.xml                                                                                                                          
    -a---        14/01/2014     18:43     136534 deployments-7978.xml                                                                                                                          
    -a---        14/01/2014     18:43      51797 deployments-7979.xml                                                                                                                          
    -a---        14/01/2014     18:43     136534 deployments-7980.xml                                                                                                                          
    -a---        14/01/2014     18:43       6305 deployments-8001.xml                                                                                                                          
    
    

    Now I can search the files to find what I need.

    In case anyone else wants to do this, here's the code I used.
    function Export-DeploymentLogs {
      param (
        [String] $TargetDirectory = '.'
      )
    
      $AttachmentList = Invoke-RestMethod -Uri 'http://localhost:10300/static?start=0&pagesize=999999999'
    
      $AttachmentList |
      Where {
        $_.Key -like '*deployment*'
      } |
      Foreach {
        $AttachmentName = $_.Key.Split('/')[1]
    
        Invoke-WebRequest -Uri "http://localhost:10300/static/$($_.Key)" -OutFile "$TargetDirectory\$AttachmentName.xml.gz"
    
        gzip.exe --decompress "$TargetDirectory\$AttachmentName.xml.gz"
      }
    }
    

    It requires the gzip utlity to be on the path. I use version 1.2.4. You can download it from http://www.gzip.org/.

    RavenDB pages the links to attachments through the static resource. Starting at 0 with the pagesize set larger than the number of attachments in the database, the static endpoint returns a link to all the attachments in the database.

    The key property contains the word 'deployment' if the attachment is a deployment log. The function filters on this to ignore other types of attachment.

    The full key is a relative URL for the attachment. If you fetch it, you get a gzip-compressed XML file.

    gzip.exe deletes the input file and writes the decompressed version with an .xml extension.

    In the end, the target directory contains a bunch of XML files.
    Iain Elder, Skyscanner
  • Awesome, thanks for posting back your solution for everyone else to use if they need it!
    Systems Software Engineer

    Redgate Software

  • I added this script to a new Bitbucket repo called "RGDM Tools".

    I'll copy the rest of my posted code there too.

    Thanks again!
    Iain Elder, Skyscanner
Sign In or Register to comment.