Introduction:
In this article , we will demonstrate how to export Cloudhub logs automatically based on time interval using Mule 4.
Use Case:
In Mulesoft for lower subscriptions plan , cloudhub application logs can be access up to 100MB or 30 days.Once this limit reaches, we can’t access the logs.
Inorder to overcome this limit, we can go for upgrading higher subscriptions plan in Mulesoft or using third-party logging and monitoring applications like Splunk, but these approaches are not available for free.
Cloudhub itself provide APIs to download log-file for free of cost.In this article, we will see step-by-step procedure to integrate Cloudhub API to download log-file using Mule4 and store it in our system or external document storage applications.
Architecture Diagram:
Implementation:
i) logs-export-flow:
This flow will be intitator flow which will run everyday at 12AM UTC.We need to provide Application Name and EnivironmentID in property file for which logs needs to be exported.
This flow will iterate based on Application Name and calls inception flow.
Note: If you want to export log file for more than one application at a time, then provide the application names as comma separated values in anypointApplication as shown below
configurationproperty file:
logs-export-flow.xml:
<flow name=”logs-backup-flow” doc:id=”21072988-d26a-43d9-bb19-bb3a2eaa2532″ >
<scheduler doc:name=”Scheduler” doc:id=”077ba982-5dce-42a6-94f5-beae710f6c8a” >
<scheduling-strategy >
<cron expression=”0 0 0 ? * * *” timeZone=”UTC” />
</scheduling-strategy>
</scheduler>
<ee:transform doc:name=”Transform Message” doc:id=”31b2df93-4b73-4144-b0da-c90170f37083″ >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/java
—
p(‘anypointApplication’) default ” splitBy ‘,’ ]]></ee:set-payload>
</ee:message>
</ee:transform>
<set–variable value=”#[payload]” doc:name=”configurationCollection” doc:id=”4da2d18d-aa71-4af5-a84a-a9fbc6574418″ variableName=”configurationCollection”/>
<set-variable value=”${anypointEnvironmentId}” doc:name=”anypointEnvironmentId” doc:id=”a976d390-f1a9-45bc-b87c-4d1e47d4e766″ variableName=”anypointEnvironmentId”/>
<foreach doc:name=”for-each iteration” doc:id=”80f64466-f03f-438b-b82c-937f2eacd245″ collection=”#[vars.configurationCollection]”>
<set–variable value=”#[vars.configurationCollection[vars.counter – 1]]” doc:name=”set config ” doc:id=”cf00c3c0-daa8-422b-a847-4f2bf7d9c26e” variableName=”config”/>
<set-variable value=”#[payload]” doc:name=”anypointApplication” doc:id=”4bcdd84a-868c-4873-b50c-24e447288f38″ variableName=”anypointApplication”/>
<flow-ref doc:name=”calling – inception-flow” doc:id=”9442ceb6-f6db-40a2-a901-f5acb80525fb” name=”inception-flow”/>
</foreach>
<error-handler >
<on-error-propagate enableNotifications=”true” logException=”true” doc:name=”On Error Propagate” doc:id=”8c312a53-2cbd-46c7-a728-e92a66302d66″ >
<ee:transform doc:name=”Transform Message” doc:id=”7adc9359-4773-4a0b-af18-721aa5838f17″ >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
—
{
code: error.errorMessage.attributes.statusCode,
message: error.detailedDescription
}]]></ee:set-payload>
</ee:message>
<ee:variables >
<ee:set-variable variableName=”httpStatus” ><![CDATA[%dw 2.0
output application/java
—
error.errorMessage.attributes.statusCode default 500]]></ee:set-variable>
</ee:variables>
</ee:transform>
</on-error-propagate>
</error-handler>
</flow>
To find Cloudhub Environment Id, follow the below steps:
i) Log into Cloudhub Account.
ii) Go to Access Management and Select Environment.
iii) Hover over the environment name and the URL that shows the environment ID at the end.
ii) Inception flow
This flow is used to call “get-instancesID” and “download_and_zip_cloudhub_logs” flows.
inception-flow.xml:
<sub-flow name=”inception-flow” doc:id=”0c851d1b-b8bd-4bac-a50a-9bead27e6a16″ >
<flow-ref doc:name=”calling_get-instancesID-flow” doc:id=”a068787b-65a1-4b7a-b6d2-de6a34c2a461″ name=”get-instancesID”/>
<flow-ref doc:name=”calling_download_and_zip_cloudhub_logs_flow” doc:id=”00765964-60f7-45e2-9f81-b581dde5f6e6″ name=”download_and_zip_cloudhub_logs” />
</sub-flow>
iii) get-instancesID Flow:
This flow is used to get instaceId of applications for which logs need to be exported.
get-instancesID.xml
<sub-flow name=”get-instancesID” doc:id=”48ee29c2-efa2-435e-9030-93f51ebe62b5″ >
<set-payload value=”{}” doc:name=”Set Payload” doc:id=”48870e95-d3c9-4bee-856a-c7f09b8e329a” />
<until-successful maxRetries=”5″ doc:name=”Until Successful” doc:id=”caf5181f-6502-4ed7-a83d-ec479896108c” >
<http:request method=”GET” doc:name=”get instances” doc:id=”421325ee-33fa-46a6-bdd7-b0932b664756″ config-ref=”Cloudhub_Config” path=”/v2/applications/{domain}/deployments” outputMimeType=”application/json”>
<http:headers><![CDATA[#[output application/java
—
{
“X-ANYPNT-ENV-ID” : vars.anypointEnvironmentId
}]]]></http:headers>
<http:uri-params><![CDATA[#[output application/java
—
{
“domain” : vars.anypointApplication
}]]]></http:uri-params>
</http:request>
</until-successful>
<ee:transform doc:name=”payload transformation” doc:id=”6a0c40a9-0853-4491-8bc1-4609ebe82bf5″>
<ee:message>
<ee:set-payload><![CDATA[%dw 2.0
output application/java
—
payload]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level=”INFO” doc:name=”Logger” doc:id=”464220f0-0a4d-4720-a2b0-e58d05658a1b” message=”#[payload.data[0].instances]”/>
<set-variable value=”#[payload.data[0].instances.instanceId[0]]” doc:name=”set instancesID” doc:id=”369989b3-e41e-472d-abed-751414a95e5f” variableName=”instancesID” />
</sub-flow>
iv) download_and_zip_cloudhub_logs:
This flow is used to download log-files from cloudhub, compress, zip it and save it in our system.
download_and_zip_cloudhub_logs.xml
<sub-flow name=”download_and_zip_cloudhub_logs” doc:id=”5fc8a173-c2b7-461a-8259-ac3f4f1a7497″ >
<set-payload value=”{}” doc:name=”Set Payload” doc:id=”4ca2cee4-137e-4277-8c97-de89d4247d97″ />
<ee:transform doc:name=”Transform Message” doc:id=”75d5fd26-15ba-4869-99c3-a2584ee0d899″ >
<ee:message >
</ee:message>
<ee:variables >
<ee:set–variable variableName=”domain” ><![CDATA[%dw 2.0
output application/java
—
vars.config]]></ee:set-variable>
<ee:set–variable variableName=”instanceId” ><![CDATA[%dw 2.0
output application/java
—
vars.instancesID]]></ee:set-variable>
</ee:variables>
</ee:transform>
<until-successful maxRetries=”5″ doc:name=”Until Successful” doc:id=”1fd35258-fe2d-4c1e-822e-a2215d32c6be” >
<http:request method=”GET” doc:name=”retrieve cloudhub logs” doc:id=”3075bd39-fae3-4644-b832-421ddc488cbb” config-ref=”Cloudhub_Config” path=”/v2/applications/{domain}/instances/{instanceId}/log-file” outputMimeType=”application/json”>
<http:headers><![CDATA[#[output application/java
—
{
“X-ANYPNT-ENV-ID” : vars.anypointEnvironmentId
}]]]></http:headers>
<http:uri-params><![CDATA[#[output application/java
—
{
“domain” : vars.anypointApplication,
“instanceId” : vars.instancesID
}]]]></http:uri-params>
</http:request>
</until-successful>
<compression:compress doc:name=”compress and zip logs” doc:id=”a62dcba5-21e5-40d1-a322-50b3a3c5911b” >
<compression:compressor >
<compression:zip-compressor />
</compression:compressor>
</compression:compress>
<set-variable value=”GIVE PATH WHERE FILE NEEDS TO BE DOWNLOADED” doc:name=”pathname” doc:id=”711ad7bc-e180-4e94-a9c5-20a66466e16e” variableName=”pathname”/>
<set–variable value=”#[vars.anypointApplication]” doc:name=”filename” doc:id=”49f908ed-2190-4d1a-ba7c-21e543a1722f” variableName=”filename”/>
<set-variable value=”#[now() as Date as String]” doc:name=”timestamp” doc:id=”42f8b096-00ba-4ee4-8796-e50cff1728c0″ variableName=”timestamp”/>
<set–variable value=’#[vars.pathname ++ vars.filename ++ “_” ++ vars.timestamp ++”.zip”]’ doc:name=”filepath” doc:id=”4ba9bac0-e71a-4ca0-8cbf-8d2f7b733f24″ variableName=”filepath”/>
<file:write doc:name=”Write files in local drive” doc:id=”232c75fe-189a-4a16-ace0-3239de22909e” config-ref=”File_Config” mode=”CREATE_NEW” path=”#[vars.filepath]”/>
<logger level=”INFO” doc:name=”Logger” doc:id=”32977ba3-cf19-4499-933d-bdc032ee34ff” message=”Log file written succesfully”/>
</sub-flow>
Conclusion:
Cloudhub logs are downloaded and stored in the file path specified with naming convention (Applicationname_datestamp). We can also export logfile to FTP, SFTP, Amazon S3 or any other document Storage applications.
Reference link:
For additional input please check below
