It is recommended to quickly review exercise 3 from homework 1 before starting.
mimetext/htmlrootassigneelast_run_timestampA;9qfMpersist_js_state·has_pluto_hook_features§cell_id$8b519a7d-b64f-4c0c-8d74-17a963cb381ddepends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$0fbd2560-1e4c-45b7-89a6-d2b622f9f907queued¤logsrunning¦outputbody2plot_loss_surface (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA;9rUeðpersist_js_state·has_pluto_hook_features§cell_id$0fbd2560-1e4c-45b7-89a6-d2b622f9f907depends_on_disabled_cells§runtime )published_object_keysdepends_on_skipped_cells§errored$ff5d316f-806c-4652-97d8-323462395c69queued¤logsrunning¦outputbodyٚ
Submission by: Mara Mustermann (m.mustermann@campus.tu-berlin.de, Matr.-Nr.: 456123)
mimetext/htmlrootassigneelast_run_timestampA;9qӰpersist_js_state·has_pluto_hook_features§cell_id$ff5d316f-806c-4652-97d8-323462395c69depends_on_disabled_cells§runtimeUpublished_object_keysdepends_on_skipped_cells§errored$e5e014e6-e24d-4de0-a216-61c0f9d9c331queued¤logsrunning¦outputbodymsgUndefVarError: `dt` not definedstacktracecall_shorttop-level scopeinlined£urlpathu/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#e5e014e6-e24d-4de0-a216-61c0f9d9c331source_packagecalltop-level scopelinfo_typeCore.CodeInfolinefile:H3_Custom_Types.jl#==#e5e014e6-e24d-4de0-a216-61c0f9d9c331functop-level scopeparent_modulefrom_c«plain_errorٲUndefVarError: `dt` not defined
Stacktrace:
[1] top-level scope
@ ~/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#e5e014e6-e24d-4de0-a216-61c0f9d9c331:1mime'application/vnd.pluto.stacktrace+objectrootassigneelast_run_timestampA;9r -ϰpersist_js_state·has_pluto_hook_features§cell_id$e5e014e6-e24d-4de0-a216-61c0f9d9c331depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$02a62f7b-6014-439a-a254-49f53f204abbqueued¤logsrunning¦outputbodyْ
As we can see in this plot, our original data didn't come from a normal distribution. Standardization transforms the data such that $\mu = 0$ and $\sigma=1$, but doesn't make it normally distributed.
mimetext/htmlrootassigneelast_run_timestampA;9fޘpersist_js_state·has_pluto_hook_features§cell_id$bcdd297b-c52c-4076-9329-929d69b8021bdepends_on_disabled_cells§runtime ᰵpublished_object_keysdepends_on_skipped_cells§errored$3f81ccce-5744-42ba-aa1e-b3da6212b9efqueued¤logsrunning¦outputbodymsg'UndefVarError: `optimizers` not definedstacktracecall_shorttop-level scopeinlined£urlpathu/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#3f81ccce-5744-42ba-aa1e-b3da6212b9efsource_packagecalltop-level scopelinfo_typeCore.CodeInfolinefile:H3_Custom_Types.jl#==#3f81ccce-5744-42ba-aa1e-b3da6212b9effunctop-level scopeparent_modulefrom_c«plain_errorٺUndefVarError: `optimizers` not defined
Stacktrace:
[1] top-level scope
@ ~/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#3f81ccce-5744-42ba-aa1e-b3da6212b9ef:1mime'application/vnd.pluto.stacktrace+objectrootassigneelast_run_timestampA;9sJ8persist_js_state·has_pluto_hook_features§cell_id$3f81ccce-5744-42ba-aa1e-b3da6212b9efdepends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$a3634333-1149-4392-a7e9-350a56d67d2bqueued¤logsrunning¦outputbody0plot_trajectory (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA;9s:*persist_js_state·has_pluto_hook_features§cell_id$a3634333-1149-4392-a7e9-350a56d67d2bdepends_on_disabled_cells§runtime =published_object_keysdepends_on_skipped_cells§errored$be2c5285-fbd8-4c37-8b17-db6db95c8a4equeued¤logsrunning¦outputbody
Task (2 points)
Now define two functions transform and inverse_transform:
A function transform that takes an array and a Standardize object and implements the standardization
$$X' = \frac{X-\mu}{\sigma} \quad .$$
A function inverse_transform that takes an array and a Standardize object and implements the inverse standardization
Click plot_trajectory below to reveal the code that produced the plot. The most relevant part is the inner loop where the optimizer is called:
state = Optimisers.setup(opt, params)
for i in 1:steps
grad = ... # Compute gradient. We will learn how this works next week.
state, params = Optimisers.update(state, params, grad)
end
Make sure that you define a struct called GradientDescent.
mimetext/htmlrootassigneelast_run_timestampA;9rB
persist_js_state·has_pluto_hook_features§cell_id$4eac0e3f-7212-4038-a24e-8d6ca3fbe659depends_on_disabled_cells§runtime "published_object_keysdepends_on_skipped_cells§errored$3d638561-6760-4260-991a-b0b0c77745a3queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA;9ppersist_js_state·has_pluto_hook_features§cell_id$3d638561-6760-4260-991a-b0b0c77745a3depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$17ff11e9-84f8-4911-845b-930a53c73e0equeued¤logsrunning¦outputbodymsgUndefVarError: `dt` not definedstacktracecall_shorttop-level scopeinlined£urlpathu/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#17ff11e9-84f8-4911-845b-930a53c73e0esource_packagecalltop-level scopelinfo_typeCore.CodeInfolinefile:H3_Custom_Types.jl#==#17ff11e9-84f8-4911-845b-930a53c73e0efunctop-level scopeparent_modulefrom_c«plain_errorٲUndefVarError: `dt` not defined
Stacktrace:
[1] top-level scope
@ ~/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#17ff11e9-84f8-4911-845b-930a53c73e0e:1mime'application/vnd.pluto.stacktrace+objectrootassigneelast_run_timestampA;9r"Kpersist_js_state·has_pluto_hook_features§cell_id$17ff11e9-84f8-4911-845b-930a53c73e0edepends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$3ac8faba-1af9-4fc6-a1d0-37225b9ebb02queued¤logsrunning¦outputbody
Exercise 2 – Gradient-based optimization
As we have seen in the previous exercise, strucs / composite types are a common way to implement algorithms. In this exercise, we will extend the gradient-based optimization algorithms from Optimisers.jl.
mimetext/htmlrootassigneelast_run_timestampA;9fŰpersist_js_state·has_pluto_hook_features§cell_id$3ac8faba-1af9-4fc6-a1d0-37225b9ebb02depends_on_disabled_cells§runtime tpublished_object_keysdepends_on_skipped_cells§errored$bb0d58f9-380e-43fa-9524-0108a2816884queued¤logsrunning¦outputbodymsg(UndefVarError: `Standardize` not definedstacktracecall_shorttop-level scopeinlined£urlpathu/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#bb0d58f9-380e-43fa-9524-0108a2816884source_packagecalltop-level scopelinfo_typeCore.CodeInfolinefile:H3_Custom_Types.jl#==#bb0d58f9-380e-43fa-9524-0108a2816884functop-level scopeparent_modulefrom_c«plain_errorٻUndefVarError: `Standardize` not defined
Stacktrace:
[1] top-level scope
@ ~/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#bb0d58f9-380e-43fa-9524-0108a2816884:1mime'application/vnd.pluto.stacktrace+objectrootassigneelast_run_timestampA;9r-persist_js_state·has_pluto_hook_features§cell_id$bb0d58f9-380e-43fa-9524-0108a2816884depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$5061a130-fc0a-4306-bdf5-6966e8de938aqueued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA;9qfopersist_js_state·has_pluto_hook_features§cell_id$5061a130-fc0a-4306-bdf5-6966e8de938adepends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$ddd6e83e-5a0d-4ff0-afe4-dedfc860994cqueued¤logsrunning¦outputbodyR
Student information
mimetext/htmlrootassigneelast_run_timestampA;9fqpersist_js_state·has_pluto_hook_features§cell_id$ddd6e83e-5a0d-4ff0-afe4-dedfc860994cdepends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$f9192279-9d12-485e-bb9d-650be2dbd878queued¤logsrunning¦outputbodyTmimetext/htmlrootassigneelast_run_timestampA;9spersist_js_state·has_pluto_hook_features§cell_id$f9192279-9d12-485e-bb9d-650be2dbd878depends_on_disabled_cells§runtime
˵published_object_keysdepends_on_skipped_cells§errored$78fcd02f-1a6e-4799-b042-4542a1ec7847queued¤logsrunning¦outputbodymsgUndefVarError: `dt` not definedstacktracecall_shorttop-level scopeinlined£urlpathu/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#78fcd02f-1a6e-4799-b042-4542a1ec7847source_packagecalltop-level scopelinfo_typeCore.CodeInfolinefile:H3_Custom_Types.jl#==#78fcd02f-1a6e-4799-b042-4542a1ec7847functop-level scopeparent_modulefrom_c«plain_errorٲUndefVarError: `dt` not defined
Stacktrace:
[1] top-level scope
@ ~/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#78fcd02f-1a6e-4799-b042-4542a1ec7847:1mime'application/vnd.pluto.stacktrace+objectrootassigneelast_run_timestampA;9r? persist_js_state·has_pluto_hook_features§cell_id$78fcd02f-1a6e-4799-b042-4542a1ec7847depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$2341dd4a-a35a-4a62-815a-e972d04844abqueued¤logsrunning¦outputbody
Let's plot the trajectory of our optimizers on the "loss surface".
Add as many optimizers and sliders as you want, change the initial parameters and the number of steps taken.
In this exercise, we will refactor the standardization code from homework 1 by using custom types. Let's load the dataset from last time:
mimetext/htmlrootassigneelast_run_timestampA;9f˰persist_js_state·has_pluto_hook_features§cell_id$d36914f8-f7d4-407a-a842-436f4035157bdepends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$3b65f124-75c2-4fb3-8b59-94015500acbfqueued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA;9rO$ypersist_js_state·has_pluto_hook_features§cell_id$3b65f124-75c2-4fb3-8b59-94015500acbfdepends_on_disabled_cells§runtimeNpublished_object_keysdepends_on_skipped_cells§errored$755b8685-0711-48a2-a3eb-f80af39f10e1queued¤logsrunning¦outputbodyprefixPlots.PlotlyBackendelementstypestructprefix_shortPlotlyBackendobjectidffffffffb3054924mime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA;9p魧persist_js_state·has_pluto_hook_features§cell_id$755b8685-0711-48a2-a3eb-f80af39f10e1depends_on_disabled_cells§runtimeΕ(lpublished_object_keysdepends_on_skipped_cells§errored$3bb76227-82cd-4924-bfb3-1b8504e2dc44queued¤logsrunning¦outputbody%loss (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA;9rJ1persist_js_state·has_pluto_hook_features§cell_id$3bb76227-82cd-4924-bfb3-1b8504e2dc44depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$5dc7d04d-f6a5-4265-b890-d74ea4b3c602queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA;9fpersist_js_state·has_pluto_hook_features§cell_id$5dc7d04d-f6a5-4265-b890-d74ea4b3c602depends_on_disabled_cells§runtime$published_object_keysdepends_on_skipped_cells§errored$fd30da51-0890-4ebe-8b14-8975ec12fec0queued¤logsrunning¦outputbody
Keep working on it!
Make sure that you define a struct called Standardize.
Define an Optimisers.init function for GradientDescent that returns nothing.
We return nothing since gradient descent doesn't require a state.
mimetext/htmlrootassigneelast_run_timestampA;9rApersist_js_state·has_pluto_hook_features§cell_id$b0d679ba-13f8-4e70-ac42-68acb2ff83d1depends_on_disabled_cells§runtime :published_object_keysdepends_on_skipped_cells§errored$f83196a4-a0fd-44a3-b3d1-29730599830dqueued¤logsrunning¦outputbodymsgUndefVarError: `dt` not definedstacktracecall_shorttop-level scopeinlined£urlpathu/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#f83196a4-a0fd-44a3-b3d1-29730599830dsource_packagecalltop-level scopelinfo_typeCore.CodeInfolinefile:H3_Custom_Types.jl#==#f83196a4-a0fd-44a3-b3d1-29730599830dfunctop-level scopeparent_modulefrom_c«plain_errorٲUndefVarError: `dt` not defined
Stacktrace:
[1] top-level scope
@ ~/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#f83196a4-a0fd-44a3-b3d1-29730599830d:1mime'application/vnd.pluto.stacktrace+objectrootassigneetransformed_datalast_run_timestampA;9r2Apersist_js_state·has_pluto_hook_features§cell_id$f83196a4-a0fd-44a3-b3d1-29730599830ddepends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$6d5bf1e5-5ab0-4351-89d0-ddb6fcfa27b6queued¤logsrunning¦outputbody
Keep working on it!
Make sure that you define a struct called GradientDescent.
mimetext/htmlrootassigneelast_run_timestampA;9rAapersist_js_state·has_pluto_hook_features§cell_id$6d5bf1e5-5ab0-4351-89d0-ddb6fcfa27b6depends_on_disabled_cells§runtime X published_object_keysdepends_on_skipped_cells§errored$822d03c6-48f8-47b0-8f25-b2de45e4c589queued¤logsrunning¦outputbodys
Optional task
Implement the Adam optimizer, which was introduced in this paper by Kingma and Ba. Add it to the visualization above and compare it to Optimizer.jl's Adam implementation.
Statefulness could also have been implemented by making the state a field in the struct of the optimizer and/or making the struct mutable.
However, mutable structs are usually less performant. The dimensionality of the state also often depends on the dimensionality of the input parameters we optimize over.
Reminder: By convention, functions that modify their inputs in-place are given an exclamation point ! at the end of their name.
mimetext/htmlrootassigneelast_run_timestampA;9r@ݵpersist_js_state·has_pluto_hook_features§cell_id$d5052369-0b47-42e7-b661-058c7365c032depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$cf3c52c7-fc5e-4191-be9d-394660385801queued¤logsrunning¦outputbodymsg(UndefVarError: `Standardize` not definedstacktracecall_shorttop-level scopeinlined£urlpathu/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#cf3c52c7-fc5e-4191-be9d-394660385801source_packagecalltop-level scopelinfo_typeCore.CodeInfolinefile:H3_Custom_Types.jl#==#cf3c52c7-fc5e-4191-be9d-394660385801functop-level scopeparent_modulefrom_c«plain_errorٻUndefVarError: `Standardize` not defined
Stacktrace:
[1] top-level scope
@ ~/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#cf3c52c7-fc5e-4191-be9d-394660385801:1mime'application/vnd.pluto.stacktrace+objectrootassigneedtlast_run_timestampA;9r ]persist_js_state·has_pluto_hook_features§cell_id$cf3c52c7-fc5e-4191-be9d-394660385801depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$2cd9cef0-d746-411a-861f-b8a035f4a1d3queued¤logsrunning¦outputbodymsg-UndefVarError: `transformed_data` not definedstacktracecall_shorttop-level scopeinlined£urlpathu/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#2cd9cef0-d746-411a-861f-b8a035f4a1d3source_packagecalltop-level scopelinfo_typeCore.CodeInfolinefile:H3_Custom_Types.jl#==#2cd9cef0-d746-411a-861f-b8a035f4a1d3functop-level scopeparent_modulefrom_c«plain_errorUndefVarError: `transformed_data` not defined
Stacktrace:
[1] top-level scope
@ ~/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#2cd9cef0-d746-411a-861f-b8a035f4a1d3:1mime'application/vnd.pluto.stacktrace+objectrootassigneelast_run_timestampA;9r5persist_js_state·has_pluto_hook_features§cell_id$2cd9cef0-d746-411a-861f-b8a035f4a1d3depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$d15eb189-6e4d-4c70-a1cb-159e6a4eb8ebqueued¤logsrunning¦outputbodyI
Tip
Gradient descent is typically used in deep learning in combination with mini-batches, which introduce stochasticity.
This example had no stochasticity and other optimization algorithms would have been better suited.
mimetext/htmlrootassigneelast_run_timestampA;9srpersist_js_state·has_pluto_hook_features§cell_id$d15eb189-6e4d-4c70-a1cb-159e6a4eb8ebdepends_on_disabled_cells§runtime Нpublished_object_keysdepends_on_skipped_cells§errored$955a9751-38d0-4327-9a95-a0523c2d54b8queued¤logsrunning¦outputbodymsgUndefVarError: `dt` not definedstacktracecall_shorttop-level scopeinlined£urlpathu/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#955a9751-38d0-4327-9a95-a0523c2d54b8source_packagecalltop-level scopelinfo_typeCore.CodeInfolinefile:H3_Custom_Types.jl#==#955a9751-38d0-4327-9a95-a0523c2d54b8functop-level scopeparent_modulefrom_c«plain_errorٲUndefVarError: `dt` not defined
Stacktrace:
[1] top-level scope
@ ~/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#955a9751-38d0-4327-9a95-a0523c2d54b8:1mime'application/vnd.pluto.stacktrace+objectrootassigneelast_run_timestampA;9r%persist_js_state·has_pluto_hook_features§cell_id$955a9751-38d0-4327-9a95-a0523c2d54b8depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$62a1289b-ce6a-444d-8b20-92bd4fa3d05dqueued¤logsrunning¦outputbodya
For the purpose of plotting, we define a range of parameters:
mimetext/htmlrootassigneelast_run_timestampA;9f4װpersist_js_state·has_pluto_hook_features§cell_id$62a1289b-ce6a-444d-8b20-92bd4fa3d05ddepends_on_disabled_cells§runtime $published_object_keysdepends_on_skipped_cells§errored$221fd504-4cd6-4bab-b2a5-a590fff26a52queued¤logsrunning¦outputbodymsg(UndefVarError: `Standardize` not definedstacktracecall_shorttop-level scopeinlined£urlpathu/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#221fd504-4cd6-4bab-b2a5-a590fff26a52source_packagecalltop-level scopelinfo_typeCore.CodeInfolinefile:H3_Custom_Types.jl#==#221fd504-4cd6-4bab-b2a5-a590fff26a52functop-level scopeparent_modulefrom_c«plain_errorٻUndefVarError: `Standardize` not defined
Stacktrace:
[1] top-level scope
@ ~/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#221fd504-4cd6-4bab-b2a5-a590fff26a52:1mime'application/vnd.pluto.stacktrace+objectrootassigneelast_run_timestampA;9r/аpersist_js_state·has_pluto_hook_features§cell_id$221fd504-4cd6-4bab-b2a5-a590fff26a52depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$a191392f-eb62-459d-8554-901d9db29950queued¤logsrunning¦outputbodymsg-UndefVarError: `transformed_data` not definedstacktracecall_shorttop-level scopeinlined£urlpathu/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#a191392f-eb62-459d-8554-901d9db29950source_packagecalltop-level scopelinfo_typeCore.CodeInfolinefile:H3_Custom_Types.jl#==#a191392f-eb62-459d-8554-901d9db29950functop-level scopeparent_modulefrom_c«plain_errorUndefVarError: `transformed_data` not defined
Stacktrace:
[1] top-level scope
@ ~/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#a191392f-eb62-459d-8554-901d9db29950:1mime'application/vnd.pluto.stacktrace+objectrootassigneelast_run_timestampA;9r:ðpersist_js_state·has_pluto_hook_features§cell_id$a191392f-eb62-459d-8554-901d9db29950depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$67d7c60c-1c5e-4760-a9e1-4538e2ccc10equeued¤logsrunning¦outputbodyD
Initializing the optimizer
Some optimizers require a state. This state is used to save additional information about previous optimization steps, for example in optimizers that use "momentum" terms like ADAM.
When setting up an Optimizer in Optimisers.jl, init is called on it, which initializes a state. init takes two arguments: an optimizer object and an array of initial parameters.
function Optimisers.init(alg::MyOptimiser, params_init::AbstractArray)
state = ... # some computation
return state
end
Interlude: Gradient-based optimization in Deep Learning
Gradient-based optimization methods are used in deep learning, where we want to optimize a neural network $f$ that is parametrized by parameters $\theta$ (e.g. the weights and biases of $f$).
This neural network could e.g. be a classifier that predicts the output class $\hat{y}$ for some input image $x$:
$$\hat{y} = f_\theta(x)$$
In supervised learning, we are also given true labels $y$. Based on the true and predicted labels, we define a loss
$$\mathcal{L}_\theta(\hat{y}, y)$$
which depends on $\theta$, since the parameters of our neural network determine the prediction $\hat{y}$.
To optimize $\theta$, we typically compute the gradient of our loss with respect to our parameters and update the parameters using gradient-based optimization methods.
mimetext/htmlrootassigneelast_run_timestampA;9f!persist_js_state·has_pluto_hook_features§cell_id$c322fd53-78db-40ae-b7ba-612200ab6617depends_on_disabled_cells§runtime +\published_object_keysdepends_on_skipped_cells§errored$dc4f7f55-dde5-4ee6-b43d-7f64269b55d1queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA;9fSpersist_js_state·has_pluto_hook_features§cell_id$dc4f7f55-dde5-4ee6-b43d-7f64269b55d1depends_on_disabled_cells§runtime%Vpublished_object_keysdepends_on_skipped_cells§errored$cd7c244d-8a4c-4aaf-ad4b-891744e004bequeued¤logsrunning¦outputbody $
mimetext/htmlrootassigneelast_run_timestampA;9s|persist_js_state·has_pluto_hook_features§cell_id$cd7c244d-8a4c-4aaf-ad4b-891744e004bedepends_on_disabled_cells§runtimeWpublished_object_keysdepends_on_skipped_cells§errored$ed873ea2-d4bd-46e7-b89b-4e5494b606e6queued¤logsrunning¦outputbodymsg,UndefVarError: `GradientDescent` not definedstacktracecall_shorttop-level scopeinlined£urlpathu/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#ed873ea2-d4bd-46e7-b89b-4e5494b606e6source_packagecalltop-level scopelinfo_typeCore.CodeInfolinefile:H3_Custom_Types.jl#==#ed873ea2-d4bd-46e7-b89b-4e5494b606e6functop-level scopeparent_modulefrom_c«plain_errorٿUndefVarError: `GradientDescent` not defined
Stacktrace:
[1] top-level scope
@ ~/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jl#==#ed873ea2-d4bd-46e7-b89b-4e5494b606e6:1mime'application/vnd.pluto.stacktrace+objectrootassigneeoptimizerslast_run_timestampA;9spersist_js_state·has_pluto_hook_features§cell_id$ed873ea2-d4bd-46e7-b89b-4e5494b606e6depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$b3832fb6-dd19-4b38-8117-b0a11d1bc34fqueued¤logsrunning¦outputbody
Interlude: Understanding Optimisers.jl
All optimization algorithms in Optimisers.jl are sub-types of the abstract type AbstractRule.
To extend Optimisers.jl, we simply need to define a custom sub-type and overload two functions init and apply! using multiple dispatch.
The actual update step is handled automatically by Optimisers.jl. After implementing the "internal" functions init and apply! to define a custom optimizer, we can use the "public" API:
Optimisers.setup: calls init to set up the optimizer
Optimisers.update: calls apply! to compute new_grad and next_state and performs the update step params_new = params - new_grad
The function apply! is can do two things before the final update step is applied:
it can modify the gradient
it can update the optimizer state
function Optimisers.apply!(alg::MyOptimiser, state, params, grad)
grad_new = ... # some computation
state_new = ... # some computation
return state_new, grad_new
end
mimetext/htmlrootassigneelast_run_timestampA;9fݰpersist_js_state·has_pluto_hook_features§cell_id$e0705a25-a01e-43a8-8c1d-ccf0da22fca7depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored±cell_dependencies O$cf264b86-d86f-4c4a-af3b-03b34486e5ffprecedence_heuristic cell_id$cf264b86-d86f-4c4a-af3b-03b34486e5ffdownstream_cells_mapupstream_cells_map@md_strgetindex$ac1355a2-db94-4954-ad65-6887b7dbb7adprecedence_heuristic cell_id$ac1355a2-db94-4954-ad65-6887b7dbb7addownstream_cells_mapupstream_cells_maptransformed_data$f83196a4-a0fd-44a3-b3d1-29730599830dstd$3a100e93-7bba-4836-a4e6-e285aa6406c4precedence_heuristic cell_id$3a100e93-7bba-4836-a4e6-e285aa6406c4downstream_cells_maptest_data$cf3c52c7-fc5e-4191-be9d-394660385801$f83196a4-a0fd-44a3-b3d1-29730599830d$a191392f-eb62-459d-8554-901d9db29950$78fcd02f-1a6e-4799-b042-4542a1ec7847upstream_cells_map$80c5a262-e758-4075-896f-a0bc7ce1534cprecedence_heuristic cell_id$80c5a262-e758-4075-896f-a0bc7ce1534cdownstream_cells_mapupstream_cells_map@md_strgetindex$8b519a7d-b64f-4c0c-8d74-17a963cb381dprecedence_heuristic cell_id$8b519a7d-b64f-4c0c-8d74-17a963cb381ddownstream_cells_mapupstream_cells_map@md_strtipgetindex$0fbd2560-1e4c-45b7-89a6-d2b622f9f907precedence_heuristic cell_id$0fbd2560-1e4c-45b7-89a6-d2b622f9f907downstream_cells_mapplot_loss_surface$cd7c244d-8a4c-4aaf-ad4b-891744e004be$a3634333-1149-4392-a7e9-350a56d67d2bupstream_cells_mapP2$3b65f124-75c2-4fb3-8b59-94015500acbfP1$3b65f124-75c2-4fb3-8b59-94015500acbfloss$3bb76227-82cd-4924-bfb3-1b8504e2dc44contour$ff5d316f-806c-4652-97d8-323462395c69precedence_heuristic cell_id$ff5d316f-806c-4652-97d8-323462395c69downstream_cells_mapupstream_cells_map@md_strstudent$d03e4e95-faab-4ab3-ab27-81189cbd8231getindex$e5e014e6-e24d-4de0-a216-61c0f9d9c331precedence_heuristic cell_id$e5e014e6-e24d-4de0-a216-61c0f9d9c331downstream_cells_mapupstream_cells_maptypeofdt$cf3c52c7-fc5e-4191-be9d-394660385801$02a62f7b-6014-439a-a254-49f53f204abbprecedence_heuristic cell_id$02a62f7b-6014-439a-a254-49f53f204abbdownstream_cells_mapupstream_cells_map@md_strgetindex$bcdd297b-c52c-4076-9329-929d69b8021bprecedence_heuristic cell_id$bcdd297b-c52c-4076-9329-929d69b8021bdownstream_cells_mapupstream_cells_map@md_strgetindex$3f81ccce-5744-42ba-aa1e-b3da6212b9efprecedence_heuristic cell_id$3f81ccce-5744-42ba-aa1e-b3da6212b9efdownstream_cells_mapupstream_cells_mapoptimizers$ed873ea2-d4bd-46e7-b89b-4e5494b606e6plot_trajectory$a3634333-1149-4392-a7e9-350a56d67d2b$a3634333-1149-4392-a7e9-350a56d67d2bprecedence_heuristic cell_id$a3634333-1149-4392-a7e9-350a56d67d2bdownstream_cells_mapplot_trajectory$3f81ccce-5744-42ba-aa1e-b3da6212b9efupstream_cells_map:Zygote.gradientOptimisers.updateOptimisers$582cf667-b85b-4a7d-bb17-88d4776fc240loss$3bb76227-82cd-4924-bfb3-1b8504e2dc44similarZygote$755b8685-0711-48a2-a3eb-f80af39f10e1plot_loss_surface$0fbd2560-1e4c-45b7-89a6-d2b622f9f907Optimisers.setup+plot!$be2c5285-fbd8-4c37-8b17-db6db95c8a4eprecedence_heuristic cell_id$be2c5285-fbd8-4c37-8b17-db6db95c8a4edownstream_cells_mapupstream_cells_map@md_strtask$5061a130-fc0a-4306-bdf5-6966e8de938agetindex$ef9a0669-2d02-4f1b-aa99-6f6473d37c0fprecedence_heuristic cell_id$ef9a0669-2d02-4f1b-aa99-6f6473d37c0fdownstream_cells_mapupstream_cells_map@md_strtask$5061a130-fc0a-4306-bdf5-6966e8de938agetindex$aaea0bbf-868d-4fe0-b348-307449d06517precedence_heuristic cell_id$aaea0bbf-868d-4fe0-b348-307449d06517downstream_cells_mapupstream_cells_map$c080e799-b5c5-42cf-b5f1-8c36da89fc8fprecedence_heuristic cell_id$c080e799-b5c5-42cf-b5f1-8c36da89fc8fdownstream_cells_mapupstream_cells_map@md_strgetindex$569173af-9e54-49cb-b1cb-88b8cbda57a6precedence_heuristic cell_id$569173af-9e54-49cb-b1cb-88b8cbda57a6downstream_cells_mapupstream_cells_map@md_strgetindex$f60be2e0-9b43-46b5-96ef-7747ab56e164precedence_heuristic cell_id$f60be2e0-9b43-46b5-96ef-7747ab56e164downstream_cells_mapfeedbackupstream_cells_map$dc35d701-ab40-4933-b4c5-35a81b3f834aprecedence_heuristic cell_id$dc35d701-ab40-4933-b4c5-35a81b3f834adownstream_cells_maplr1$ed873ea2-d4bd-46e7-b89b-4e5494b606e6upstream_cells_mapCoreBase:PlutoRunner.create_bondPlutoRunnerCore.applicable@bindBase.getSlider$a96abf48-cacf-41f8-bb7a-3d085bdb0a62precedence_heuristic cell_id$a96abf48-cacf-41f8-bb7a-3d085bdb0a62downstream_cells_mapupstream_cells_map@md_strgetindex$58a671a1-3059-4fa9-b758-f3a76a573aa5precedence_heuristic cell_id$58a671a1-3059-4fa9-b758-f3a76a573aa5downstream_cells_mapupstream_cells_map$4eac0e3f-7212-4038-a24e-8d6ca3fbe659precedence_heuristic cell_id$4eac0e3f-7212-4038-a24e-8d6ca3fbe659downstream_cells_mapupstream_cells_map @md_strBase.getindex≈Nothing!Tuplehasmethod@isdefinedFloat64ismissingGradientDescentOptimisers$582cf667-b85b-4a7d-bb17-88d4776fc240nothingstill_missingVectorBasecorrectkeep_workingOptimisers.apply!isnothing$3d638561-6760-4260-991a-b0b0c77745a3precedence_heuristiccell_id$3d638561-6760-4260-991a-b0b0c77745a3downstream_cells_mapStatisticsupstream_cells_map$17ff11e9-84f8-4911-845b-930a53c73e0eprecedence_heuristic cell_id$17ff11e9-84f8-4911-845b-930a53c73e0edownstream_cells_mapupstream_cells_mapdt$cf3c52c7-fc5e-4191-be9d-394660385801$3ac8faba-1af9-4fc6-a1d0-37225b9ebb02precedence_heuristic cell_id$3ac8faba-1af9-4fc6-a1d0-37225b9ebb02downstream_cells_mapupstream_cells_map@md_strgetindex$bb0d58f9-380e-43fa-9524-0108a2816884precedence_heuristic cell_id$bb0d58f9-380e-43fa-9524-0108a2816884downstream_cells_maptransform$f83196a4-a0fd-44a3-b3d1-29730599830d$39063d08-d687-45a2-8367-734b12dcefea$7de85c6f-996a-4d5b-8236-271922aeb908upstream_cells_mapmissingStandardize$5061a130-fc0a-4306-bdf5-6966e8de938aprecedence_heuristic cell_id$5061a130-fc0a-4306-bdf5-6966e8de938adownstream_cells_maptask$8ece9aea-20f5-41db-95ca-08c8d4d2d4c1$2258b8c6-e8b7-4c06-9ece-53b254db6a2c$be2c5285-fbd8-4c37-8b17-db6db95c8a4e$ef9a0669-2d02-4f1b-aa99-6f6473d37c0f$b0d679ba-13f8-4e70-ac42-68acb2ff83d1$bebbf87b-cd7b-43d7-8025-938fc4c6944dmd_parseupstream_cells_mapMarkdown.parseMarkdown.AdmonitionMarkdown.MD*Markdown==$ddd6e83e-5a0d-4ff0-afe4-dedfc860994cprecedence_heuristic cell_id$ddd6e83e-5a0d-4ff0-afe4-dedfc860994cdownstream_cells_mapupstream_cells_map@md_strgetindex$f9192279-9d12-485e-bb9d-650be2dbd878precedence_heuristic cell_id$f9192279-9d12-485e-bb9d-650be2dbd878downstream_cells_maplr2$ed873ea2-d4bd-46e7-b89b-4e5494b606e6upstream_cells_mapCoreBase:PlutoRunner.create_bondPlutoRunnerCore.applicable@bindBase.getSlider$78fcd02f-1a6e-4799-b042-4542a1ec7847precedence_heuristic cell_id$78fcd02f-1a6e-4799-b042-4542a1ec7847downstream_cells_mapupstream_cells_maptest_data$3a100e93-7bba-4836-a4e6-e285aa6406c4plot_standardization$7de85c6f-996a-4d5b-8236-271922aeb908dt$cf3c52c7-fc5e-4191-be9d-394660385801$2341dd4a-a35a-4a62-815a-e972d04844abprecedence_heuristic cell_id$2341dd4a-a35a-4a62-815a-e972d04844abdownstream_cells_mapupstream_cells_map@md_strgetindex$d36914f8-f7d4-407a-a842-436f4035157bprecedence_heuristic cell_id$d36914f8-f7d4-407a-a842-436f4035157bdownstream_cells_mapupstream_cells_map@md_strgetindex$3b65f124-75c2-4fb3-8b59-94015500acbfprecedence_heuristic cell_id$3b65f124-75c2-4fb3-8b59-94015500acbfdownstream_cells_mapP2$0fbd2560-1e4c-45b7-89a6-d2b622f9f907P1$0fbd2560-1e4c-45b7-89a6-d2b622f9f907upstream_cells_maprange$755b8685-0711-48a2-a3eb-f80af39f10e1precedence_heuristiccell_id$755b8685-0711-48a2-a3eb-f80af39f10e1downstream_cells_mapZygote$a3634333-1149-4392-a7e9-350a56d67d2bPlotlyKaleidoPlutoUIPlotsPlotlyBaseLaTeXStringsPlutoTeachingToolsupstream_cells_mapplotly$3bb76227-82cd-4924-bfb3-1b8504e2dc44precedence_heuristic cell_id$3bb76227-82cd-4924-bfb3-1b8504e2dc44downstream_cells_maploss$0fbd2560-1e4c-45b7-89a6-d2b622f9f907$a3634333-1149-4392-a7e9-350a56d67d2bupstream_cells_map-^+*$5dc7d04d-f6a5-4265-b890-d74ea4b3c602precedence_heuristic cell_id$5dc7d04d-f6a5-4265-b890-d74ea4b3c602downstream_cells_mapupstream_cells_map$fd30da51-0890-4ebe-8b14-8975ec12fec0precedence_heuristic cell_id$fd30da51-0890-4ebe-8b14-8975ec12fec0downstream_cells_mapupstream_cells_map @md_strBase.getindex≈!Tuplehasmethod@isdefinedismissingstill_missingVectorBasecorrectkeep_workingIntStandardizeisnothing$8ece9aea-20f5-41db-95ca-08c8d4d2d4c1precedence_heuristic cell_id$8ece9aea-20f5-41db-95ca-08c8d4d2d4c1downstream_cells_mapupstream_cells_maptask$5061a130-fc0a-4306-bdf5-6966e8de938a$b0d679ba-13f8-4e70-ac42-68acb2ff83d1precedence_heuristic cell_id$b0d679ba-13f8-4e70-ac42-68acb2ff83d1downstream_cells_mapupstream_cells_map@md_strtask$5061a130-fc0a-4306-bdf5-6966e8de938agetindex$f83196a4-a0fd-44a3-b3d1-29730599830dprecedence_heuristic cell_id$f83196a4-a0fd-44a3-b3d1-29730599830ddownstream_cells_maptransformed_data$2cd9cef0-d746-411a-861f-b8a035f4a1d3$ac1355a2-db94-4954-ad65-6887b7dbb7ad$a191392f-eb62-459d-8554-901d9db29950upstream_cells_maptest_data$3a100e93-7bba-4836-a4e6-e285aa6406c4dt$cf3c52c7-fc5e-4191-be9d-394660385801transform$bb0d58f9-380e-43fa-9524-0108a2816884$6d5bf1e5-5ab0-4351-89d0-ddb6fcfa27b6precedence_heuristic cell_id$6d5bf1e5-5ab0-4351-89d0-ddb6fcfa27b6downstream_cells_mapupstream_cells_map@md_strBase.getindex≈!Tuplehasmethod@isdefinedFloat64GradientDescentBasecorrectkeep_workinghasfield$822d03c6-48f8-47b0-8f25-b2de45e4c589precedence_heuristic cell_id$822d03c6-48f8-47b0-8f25-b2de45e4c589downstream_cells_mapupstream_cells_map@md_strMarkdown.AdmonitionMarkdown.MDMarkdowngetindex$bebbf87b-cd7b-43d7-8025-938fc4c6944dprecedence_heuristic cell_id$bebbf87b-cd7b-43d7-8025-938fc4c6944ddownstream_cells_mapupstream_cells_map@md_strtask$5061a130-fc0a-4306-bdf5-6966e8de938agetindex$c2adc38f-84e3-486f-93d8-b88eea63a09bprecedence_heuristic cell_id$c2adc38f-84e3-486f-93d8-b88eea63a09bdownstream_cells_mapupstream_cells_map@md_strgetindex$5370c50b-a9b8-41dc-8cfe-1f0389df97c3precedence_heuristic cell_id$5370c50b-a9b8-41dc-8cfe-1f0389df97c3downstream_cells_mapupstream_cells_map@md_strgetindex$32e8ee04-4378-4377-ae47-b65e5ce70667precedence_heuristic cell_id$32e8ee04-4378-4377-ae47-b65e5ce70667downstream_cells_mapupstream_cells_map@md_strgetindex$befba318-8595-4fc4-b42e-9c9125e7abfbprecedence_heuristic cell_id$befba318-8595-4fc4-b42e-9c9125e7abfbdownstream_cells_mapupstream_cells_map@md_strtipgetindex$d5052369-0b47-42e7-b661-058c7365c032precedence_heuristic cell_id$d5052369-0b47-42e7-b661-058c7365c032downstream_cells_mapupstream_cells_map@md_strtipgetindex$cf3c52c7-fc5e-4191-be9d-394660385801precedence_heuristic cell_id$cf3c52c7-fc5e-4191-be9d-394660385801downstream_cells_mapdt$e5e014e6-e24d-4de0-a216-61c0f9d9c331$17ff11e9-84f8-4911-845b-930a53c73e0e$955a9751-38d0-4327-9a95-a0523c2d54b8$f83196a4-a0fd-44a3-b3d1-29730599830d$a191392f-eb62-459d-8554-901d9db29950$78fcd02f-1a6e-4799-b042-4542a1ec7847upstream_cells_maptest_data$3a100e93-7bba-4836-a4e6-e285aa6406c4Standardize$2cd9cef0-d746-411a-861f-b8a035f4a1d3precedence_heuristic cell_id$2cd9cef0-d746-411a-861f-b8a035f4a1d3downstream_cells_mapupstream_cells_maptransformed_data$f83196a4-a0fd-44a3-b3d1-29730599830dmean$d15eb189-6e4d-4c70-a1cb-159e6a4eb8ebprecedence_heuristic cell_id$d15eb189-6e4d-4c70-a1cb-159e6a4eb8ebdownstream_cells_mapupstream_cells_map@md_strtipgetindex$955a9751-38d0-4327-9a95-a0523c2d54b8precedence_heuristic cell_id$955a9751-38d0-4327-9a95-a0523c2d54b8downstream_cells_mapupstream_cells_mapdt$cf3c52c7-fc5e-4191-be9d-394660385801$62a1289b-ce6a-444d-8b20-92bd4fa3d05dprecedence_heuristic cell_id$62a1289b-ce6a-444d-8b20-92bd4fa3d05ddownstream_cells_mapupstream_cells_map@md_strgetindex$221fd504-4cd6-4bab-b2a5-a590fff26a52precedence_heuristic cell_id$221fd504-4cd6-4bab-b2a5-a590fff26a52downstream_cells_mapinverse_transform$a191392f-eb62-459d-8554-901d9db29950$39063d08-d687-45a2-8367-734b12dcefeaupstream_cells_mapmissingStandardize$a191392f-eb62-459d-8554-901d9db29950precedence_heuristic cell_id$a191392f-eb62-459d-8554-901d9db29950downstream_cells_mapupstream_cells_map≈test_data$3a100e93-7bba-4836-a4e6-e285aa6406c4inverse_transform$221fd504-4cd6-4bab-b2a5-a590fff26a52transformed_data$f83196a4-a0fd-44a3-b3d1-29730599830ddt$cf3c52c7-fc5e-4191-be9d-394660385801$67d7c60c-1c5e-4760-a9e1-4538e2ccc10eprecedence_heuristic cell_id$67d7c60c-1c5e-4760-a9e1-4538e2ccc10edownstream_cells_mapupstream_cells_map@md_strgetindex$d16b1a65-5da9-46ac-b0b8-59e55ced37f3precedence_heuristic cell_id$d16b1a65-5da9-46ac-b0b8-59e55ced37f3downstream_cells_mapupstream_cells_map@md_strgetindex$7de85c6f-996a-4d5b-8236-271922aeb908precedence_heuristic cell_id$7de85c6f-996a-4d5b-8236-271922aeb908downstream_cells_mapplot_standardization$78fcd02f-1a6e-4799-b042-4542a1ec7847upstream_cells_maphistogramexphistogram!/^πplot!*Standardizetransform$bb0d58f9-380e-43fa-9524-0108a2816884$edb7814a-eddf-4c87-8857-19bb0a0c0241precedence_heuristic cell_id$edb7814a-eddf-4c87-8857-19bb0a0c0241downstream_cells_mapupstream_cells_map@md_strgetindex$2258b8c6-e8b7-4c06-9ece-53b254db6a2cprecedence_heuristic cell_id$2258b8c6-e8b7-4c06-9ece-53b254db6a2cdownstream_cells_mapupstream_cells_map@md_strtask$5061a130-fc0a-4306-bdf5-6966e8de938agetindex$8e8d9f57-0f04-4a91-b69f-273fe1fafd61precedence_heuristic cell_id$8e8d9f57-0f04-4a91-b69f-273fe1fafd61downstream_cells_mapupstream_cells_mapBaseBase.Docs.HTML@html_str$f7347c06-c1b7-11ed-3b8e-fbf167ce9cbaprecedence_heuristic cell_id$f7347c06-c1b7-11ed-3b8e-fbf167ce9cbadownstream_cells_mapupstream_cells_mapBaseBase.Docs.HTML@html_str$c322fd53-78db-40ae-b7ba-612200ab6617precedence_heuristic cell_id$c322fd53-78db-40ae-b7ba-612200ab6617downstream_cells_mapupstream_cells_map@md_strgetindex$dc4f7f55-dde5-4ee6-b43d-7f64269b55d1precedence_heuristic cell_id$dc4f7f55-dde5-4ee6-b43d-7f64269b55d1downstream_cells_mapupstream_cells_map$cd7c244d-8a4c-4aaf-ad4b-891744e004beprecedence_heuristic cell_id$cd7c244d-8a4c-4aaf-ad4b-891744e004bedownstream_cells_mapupstream_cells_mapplot_loss_surface$0fbd2560-1e4c-45b7-89a6-d2b622f9f907$ed873ea2-d4bd-46e7-b89b-4e5494b606e6precedence_heuristic cell_id$ed873ea2-d4bd-46e7-b89b-4e5494b606e6downstream_cells_mapoptimizers$3f81ccce-5744-42ba-aa1e-b3da6212b9efupstream_cells_mapDictADAMGradientDescentMomentum=>lr1$dc35d701-ab40-4933-b4c5-35a81b3f834alr2$f9192279-9d12-485e-bb9d-650be2dbd878$b3832fb6-dd19-4b38-8117-b0a11d1bc34fprecedence_heuristic cell_id$b3832fb6-dd19-4b38-8117-b0a11d1bc34fdownstream_cells_mapupstream_cells_map@md_strgetindex$bdcb27c5-0603-49ac-b831-d78c558b31f0precedence_heuristic cell_id$bdcb27c5-0603-49ac-b831-d78c558b31f0downstream_cells_mapupstream_cells_map@md_strgetindex$33866006-11d7-46cc-870b-fec519a2b541precedence_heuristic cell_id$33866006-11d7-46cc-870b-fec519a2b541downstream_cells_mapupstream_cells_map@md_strtipgetindex$44ec9e94-f6af-431e-8841-bae44431dfa3precedence_heuristic cell_id$44ec9e94-f6af-431e-8841-bae44431dfa3downstream_cells_mapupstream_cells_map@md_strisacorrectstill_missing!student$d03e4e95-faab-4ab3-ab27-81189cbd8231==Numbergetindex$57a8108f-9ad3-4cc2-9372-bb291fc3bf86precedence_heuristic cell_id$57a8108f-9ad3-4cc2-9372-bb291fc3bf86downstream_cells_mapupstream_cells_mapTableOfContents$50b34968-a45f-4c5b-a83d-7a5b47740007precedence_heuristic cell_id$50b34968-a45f-4c5b-a83d-7a5b47740007downstream_cells_mapupstream_cells_map@md_strgetindex$06c0ad65-22d4-4c8e-ae19-4f05ba125e79precedence_heuristic cell_id$06c0ad65-22d4-4c8e-ae19-4f05ba125e79downstream_cells_mapupstream_cells_map@md_strgetindex$dcbf8963-4a5c-4e7d-81a9-3dce27fb3117precedence_heuristic cell_id$dcbf8963-4a5c-4e7d-81a9-3dce27fb3117downstream_cells_mapupstream_cells_map@md_strgetindex$5762be30-3084-4eda-9642-19dc9d4dcb10precedence_heuristic cell_id$5762be30-3084-4eda-9642-19dc9d4dcb10downstream_cells_mapupstream_cells_map@md_strBase.getindex!Tuplehasmethod@isdefinedOptimisers.initFloat64GradientDescentOptimisers$582cf667-b85b-4a7d-bb17-88d4776fc240VectorBasecorrectkeep_workingisnothing$39063d08-d687-45a2-8367-734b12dcefeaprecedence_heuristic cell_id$39063d08-d687-45a2-8367-734b12dcefeadownstream_cells_mapupstream_cells_map@md_strBase.getindexinverse_transform$221fd504-4cd6-4bab-b2a5-a590fff26a52≈!@isdefinedismissingstill_missingBasecorrectkeep_workingnot_definedStandardizetransform$bb0d58f9-380e-43fa-9524-0108a2816884isnothing$582cf667-b85b-4a7d-bb17-88d4776fc240precedence_heuristiccell_id$582cf667-b85b-4a7d-bb17-88d4776fc240downstream_cells_mapOptimisers$5762be30-3084-4eda-9642-19dc9d4dcb10$4eac0e3f-7212-4038-a24e-8d6ca3fbe659$a3634333-1149-4392-a7e9-350a56d67d2bupstream_cells_map$d03e4e95-faab-4ab3-ab27-81189cbd8231precedence_heuristic cell_id$d03e4e95-faab-4ab3-ab27-81189cbd8231downstream_cells_mapstudent$ff5d316f-806c-4652-97d8-323462395c69$44ec9e94-f6af-431e-8841-bae44431dfa3upstream_cells_map$e0705a25-a01e-43a8-8c1d-ccf0da22fca7precedence_heuristic cell_id$e0705a25-a01e-43a8-8c1d-ccf0da22fca7downstream_cells_mapupstream_cells_map@md_strgetindexcell_execution_order O$755b8685-0711-48a2-a3eb-f80af39f10e1$3d638561-6760-4260-991a-b0b0c77745a3$582cf667-b85b-4a7d-bb17-88d4776fc240$8e8d9f57-0f04-4a91-b69f-273fe1fafd61$f7347c06-c1b7-11ed-3b8e-fbf167ce9cba$bdcb27c5-0603-49ac-b831-d78c558b31f0$ddd6e83e-5a0d-4ff0-afe4-dedfc860994c$d03e4e95-faab-4ab3-ab27-81189cbd8231$ff5d316f-806c-4652-97d8-323462395c69$44ec9e94-f6af-431e-8841-bae44431dfa3$06c0ad65-22d4-4c8e-ae19-4f05ba125e79$57a8108f-9ad3-4cc2-9372-bb291fc3bf86$5061a130-fc0a-4306-bdf5-6966e8de938a$8ece9aea-20f5-41db-95ca-08c8d4d2d4c1$d36914f8-f7d4-407a-a842-436f4035157b$3a100e93-7bba-4836-a4e6-e285aa6406c4$8b519a7d-b64f-4c0c-8d74-17a963cb381d$02a62f7b-6014-439a-a254-49f53f204abb$2258b8c6-e8b7-4c06-9ece-53b254db6a2c$aaea0bbf-868d-4fe0-b348-307449d06517$80c5a262-e758-4075-896f-a0bc7ce1534c$cf3c52c7-fc5e-4191-be9d-394660385801$e5e014e6-e24d-4de0-a216-61c0f9d9c331$17ff11e9-84f8-4911-845b-930a53c73e0e$955a9751-38d0-4327-9a95-a0523c2d54b8$fd30da51-0890-4ebe-8b14-8975ec12fec0$c2adc38f-84e3-486f-93d8-b88eea63a09b$be2c5285-fbd8-4c37-8b17-db6db95c8a4e$bb0d58f9-380e-43fa-9524-0108a2816884$221fd504-4cd6-4bab-b2a5-a590fff26a52$dcbf8963-4a5c-4e7d-81a9-3dce27fb3117$f83196a4-a0fd-44a3-b3d1-29730599830d$2cd9cef0-d746-411a-861f-b8a035f4a1d3$ac1355a2-db94-4954-ad65-6887b7dbb7ad$569173af-9e54-49cb-b1cb-88b8cbda57a6$a191392f-eb62-459d-8554-901d9db29950$39063d08-d687-45a2-8367-734b12dcefea$bcdd297b-c52c-4076-9329-929d69b8021b$7de85c6f-996a-4d5b-8236-271922aeb908$78fcd02f-1a6e-4799-b042-4542a1ec7847$3ac8faba-1af9-4fc6-a1d0-37225b9ebb02$33866006-11d7-46cc-870b-fec519a2b541$c322fd53-78db-40ae-b7ba-612200ab6617$b3832fb6-dd19-4b38-8117-b0a11d1bc34f$67d7c60c-1c5e-4760-a9e1-4538e2ccc10e$befba318-8595-4fc4-b42e-9c9125e7abfb$e0705a25-a01e-43a8-8c1d-ccf0da22fca7$d5052369-0b47-42e7-b661-058c7365c032$50b34968-a45f-4c5b-a83d-7a5b47740007$d16b1a65-5da9-46ac-b0b8-59e55ced37f3$ef9a0669-2d02-4f1b-aa99-6f6473d37c0f$58a671a1-3059-4fa9-b758-f3a76a573aa5$6d5bf1e5-5ab0-4351-89d0-ddb6fcfa27b6$32e8ee04-4378-4377-ae47-b65e5ce70667$b0d679ba-13f8-4e70-ac42-68acb2ff83d1$5dc7d04d-f6a5-4265-b890-d74ea4b3c602$5762be30-3084-4eda-9642-19dc9d4dcb10$c080e799-b5c5-42cf-b5f1-8c36da89fc8f$bebbf87b-cd7b-43d7-8025-938fc4c6944d$dc4f7f55-dde5-4ee6-b43d-7f64269b55d1$4eac0e3f-7212-4038-a24e-8d6ca3fbe659$cf264b86-d86f-4c4a-af3b-03b34486e5ff$3bb76227-82cd-4924-bfb3-1b8504e2dc44$62a1289b-ce6a-444d-8b20-92bd4fa3d05d$3b65f124-75c2-4fb3-8b59-94015500acbf$0fbd2560-1e4c-45b7-89a6-d2b622f9f907$cd7c244d-8a4c-4aaf-ad4b-891744e004be$2341dd4a-a35a-4a62-815a-e972d04844ab$dc35d701-ab40-4933-b4c5-35a81b3f834a$f9192279-9d12-485e-bb9d-650be2dbd878$ed873ea2-d4bd-46e7-b89b-4e5494b606e6$a96abf48-cacf-41f8-bb7a-3d085bdb0a62$a3634333-1149-4392-a7e9-350a56d67d2b$3f81ccce-5744-42ba-aa1e-b3da6212b9ef$d15eb189-6e4d-4c70-a1cb-159e6a4eb8eb$5370c50b-a9b8-41dc-8cfe-1f0389df97c3$822d03c6-48f8-47b0-8f25-b2de45e4c589$edb7814a-eddf-4c87-8857-19bb0a0c0241$f60be2e0-9b43-46b5-96ef-7747ab56e164last_hot_reload_time shortpathH3_Custom_Types.jlprocess_statusreadypathM/home/runner/work/julia-ml-course/julia-ml-course/homework/H3_Custom_Types.jlpluto_versionv0.20.19last_save_timeA;9f;?cell_order O$8e8d9f57-0f04-4a91-b69f-273fe1fafd61$ff5d316f-806c-4652-97d8-323462395c69$f7347c06-c1b7-11ed-3b8e-fbf167ce9cba$bdcb27c5-0603-49ac-b831-d78c558b31f0$ddd6e83e-5a0d-4ff0-afe4-dedfc860994c$8ece9aea-20f5-41db-95ca-08c8d4d2d4c1$d03e4e95-faab-4ab3-ab27-81189cbd8231$44ec9e94-f6af-431e-8841-bae44431dfa3$06c0ad65-22d4-4c8e-ae19-4f05ba125e79$755b8685-0711-48a2-a3eb-f80af39f10e1$57a8108f-9ad3-4cc2-9372-bb291fc3bf86$5061a130-fc0a-4306-bdf5-6966e8de938a$d36914f8-f7d4-407a-a842-436f4035157b$3a100e93-7bba-4836-a4e6-e285aa6406c4$8b519a7d-b64f-4c0c-8d74-17a963cb381d$02a62f7b-6014-439a-a254-49f53f204abb$2258b8c6-e8b7-4c06-9ece-53b254db6a2c$3d638561-6760-4260-991a-b0b0c77745a3$aaea0bbf-868d-4fe0-b348-307449d06517$80c5a262-e758-4075-896f-a0bc7ce1534c$cf3c52c7-fc5e-4191-be9d-394660385801$e5e014e6-e24d-4de0-a216-61c0f9d9c331$17ff11e9-84f8-4911-845b-930a53c73e0e$955a9751-38d0-4327-9a95-a0523c2d54b8$fd30da51-0890-4ebe-8b14-8975ec12fec0$c2adc38f-84e3-486f-93d8-b88eea63a09b$be2c5285-fbd8-4c37-8b17-db6db95c8a4e$bb0d58f9-380e-43fa-9524-0108a2816884$221fd504-4cd6-4bab-b2a5-a590fff26a52$dcbf8963-4a5c-4e7d-81a9-3dce27fb3117$f83196a4-a0fd-44a3-b3d1-29730599830d$2cd9cef0-d746-411a-861f-b8a035f4a1d3$ac1355a2-db94-4954-ad65-6887b7dbb7ad$569173af-9e54-49cb-b1cb-88b8cbda57a6$a191392f-eb62-459d-8554-901d9db29950$39063d08-d687-45a2-8367-734b12dcefea$bcdd297b-c52c-4076-9329-929d69b8021b$78fcd02f-1a6e-4799-b042-4542a1ec7847$7de85c6f-996a-4d5b-8236-271922aeb908$3ac8faba-1af9-4fc6-a1d0-37225b9ebb02$582cf667-b85b-4a7d-bb17-88d4776fc240$33866006-11d7-46cc-870b-fec519a2b541$c322fd53-78db-40ae-b7ba-612200ab6617$b3832fb6-dd19-4b38-8117-b0a11d1bc34f$67d7c60c-1c5e-4760-a9e1-4538e2ccc10e$befba318-8595-4fc4-b42e-9c9125e7abfb$e0705a25-a01e-43a8-8c1d-ccf0da22fca7$d5052369-0b47-42e7-b661-058c7365c032$50b34968-a45f-4c5b-a83d-7a5b47740007$d16b1a65-5da9-46ac-b0b8-59e55ced37f3$ef9a0669-2d02-4f1b-aa99-6f6473d37c0f$58a671a1-3059-4fa9-b758-f3a76a573aa5$6d5bf1e5-5ab0-4351-89d0-ddb6fcfa27b6$32e8ee04-4378-4377-ae47-b65e5ce70667$b0d679ba-13f8-4e70-ac42-68acb2ff83d1$5dc7d04d-f6a5-4265-b890-d74ea4b3c602$5762be30-3084-4eda-9642-19dc9d4dcb10$c080e799-b5c5-42cf-b5f1-8c36da89fc8f$bebbf87b-cd7b-43d7-8025-938fc4c6944d$dc4f7f55-dde5-4ee6-b43d-7f64269b55d1$4eac0e3f-7212-4038-a24e-8d6ca3fbe659$cf264b86-d86f-4c4a-af3b-03b34486e5ff$3bb76227-82cd-4924-bfb3-1b8504e2dc44$62a1289b-ce6a-444d-8b20-92bd4fa3d05d$3b65f124-75c2-4fb3-8b59-94015500acbf$cd7c244d-8a4c-4aaf-ad4b-891744e004be$0fbd2560-1e4c-45b7-89a6-d2b622f9f907$2341dd4a-a35a-4a62-815a-e972d04844ab$dc35d701-ab40-4933-b4c5-35a81b3f834a$f9192279-9d12-485e-bb9d-650be2dbd878$ed873ea2-d4bd-46e7-b89b-4e5494b606e6$3f81ccce-5744-42ba-aa1e-b3da6212b9ef$a96abf48-cacf-41f8-bb7a-3d085bdb0a62$a3634333-1149-4392-a7e9-350a56d67d2b$d15eb189-6e4d-4c70-a1cb-159e6a4eb8eb$5370c50b-a9b8-41dc-8cfe-1f0389df97c3$822d03c6-48f8-47b0-8f25-b2de45e4c589$edb7814a-eddf-4c87-8857-19bb0a0c0241$f60be2e0-9b43-46b5-96ef-7747ab56e164published_objectsnbpkginstall_time_ns TQڴinstantiatedòinstalled_versionsStatisticsstdlibZygote0.7.5PlotlyKaleido2.3.0PlotlyBase0.8.20Plots1.40.11PlutoUI0.7.62LaTeXStrings1.4.0Optimisers0.4.5PlutoTeachingTools0.3.1terminal_outputsStatistics#
[0m[1mResolving...[22m
[90m===[39m
[33m[1m┌ [22m[39m[33m[1mWarning: [22m[39mPkg operation failed. Fixing stdlib dependencies and trying again...
[33m[1m└ [22m[39m[90m@ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92[39m
[0m[1mResolving...[22m
[90m===[39m
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Project.toml`
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1mPrecompiling[22m[39m packages...
27323.3 ms[32m ✓ [39mZygote
1060.8 ms[32m ✓ [39mZygote → ZygoteColorsExtZygote#
[0m[1mResolving...[22m
[90m===[39m
[33m[1m┌ [22m[39m[33m[1mWarning: [22m[39mPkg operation failed. Fixing stdlib dependencies and trying again...
[33m[1m└ [22m[39m[90m@ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92[39m
[0m[1mResolving...[22m
[90m===[39m
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Project.toml`
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1mPrecompiling[22m[39m packages...
27323.3 ms[32m ✓ [39mZygote
1060.8 ms[32m ✓ [39mZygote → ZygoteColorsExtnbpkg_sync#
[0m[1mResolving...[22m
[90m===[39m
[33m[1m┌ [22m[39m[33m[1mWarning: [22m[39mPkg operation failed. Fixing stdlib dependencies and trying again...
[33m[1m└ [22m[39m[90m@ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92[39m
[0m[1mResolving...[22m
[90m===[39m
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Project.toml`
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1mPrecompiling[22m[39m packages...
27323.3 ms[32m ✓ [39mZygote
1060.8 ms[32m ✓ [39mZygote → ZygoteColorsExtPlotlyKaleido#
[0m[1mResolving...[22m
[90m===[39m
[33m[1m┌ [22m[39m[33m[1mWarning: [22m[39mPkg operation failed. Fixing stdlib dependencies and trying again...
[33m[1m└ [22m[39m[90m@ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92[39m
[0m[1mResolving...[22m
[90m===[39m
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Project.toml`
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1mPrecompiling[22m[39m packages...
27323.3 ms[32m ✓ [39mZygote
1060.8 ms[32m ✓ [39mZygote → ZygoteColorsExtPlutoUI#
[0m[1mResolving...[22m
[90m===[39m
[33m[1m┌ [22m[39m[33m[1mWarning: [22m[39mPkg operation failed. Fixing stdlib dependencies and trying again...
[33m[1m└ [22m[39m[90m@ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92[39m
[0m[1mResolving...[22m
[90m===[39m
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Project.toml`
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1mPrecompiling[22m[39m packages...
27323.3 ms[32m ✓ [39mZygote
1060.8 ms[32m ✓ [39mZygote → ZygoteColorsExtPlots#
[0m[1mResolving...[22m
[90m===[39m
[33m[1m┌ [22m[39m[33m[1mWarning: [22m[39mPkg operation failed. Fixing stdlib dependencies and trying again...
[33m[1m└ [22m[39m[90m@ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92[39m
[0m[1mResolving...[22m
[90m===[39m
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Project.toml`
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1mPrecompiling[22m[39m packages...
27323.3 ms[32m ✓ [39mZygote
1060.8 ms[32m ✓ [39mZygote → ZygoteColorsExtPlotlyBase#
[0m[1mResolving...[22m
[90m===[39m
[33m[1m┌ [22m[39m[33m[1mWarning: [22m[39mPkg operation failed. Fixing stdlib dependencies and trying again...
[33m[1m└ [22m[39m[90m@ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92[39m
[0m[1mResolving...[22m
[90m===[39m
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Project.toml`
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1mPrecompiling[22m[39m packages...
27323.3 ms[32m ✓ [39mZygote
1060.8 ms[32m ✓ [39mZygote → ZygoteColorsExtLaTeXStrings#
[0m[1mResolving...[22m
[90m===[39m
[33m[1m┌ [22m[39m[33m[1mWarning: [22m[39mPkg operation failed. Fixing stdlib dependencies and trying again...
[33m[1m└ [22m[39m[90m@ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92[39m
[0m[1mResolving...[22m
[90m===[39m
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Project.toml`
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1mPrecompiling[22m[39m packages...
27323.3 ms[32m ✓ [39mZygote
1060.8 ms[32m ✓ [39mZygote → ZygoteColorsExtOptimisers#
[0m[1mResolving...[22m
[90m===[39m
[33m[1m┌ [22m[39m[33m[1mWarning: [22m[39mPkg operation failed. Fixing stdlib dependencies and trying again...
[33m[1m└ [22m[39m[90m@ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92[39m
[0m[1mResolving...[22m
[90m===[39m
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Project.toml`
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1mPrecompiling[22m[39m packages...
27323.3 ms[32m ✓ [39mZygote
1060.8 ms[32m ✓ [39mZygote → ZygoteColorsExtPlutoTeachingTools#
[0m[1mResolving...[22m
[90m===[39m
[33m[1m┌ [22m[39m[33m[1mWarning: [22m[39mPkg operation failed. Fixing stdlib dependencies and trying again...
[33m[1m└ [22m[39m[90m@ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92[39m
[0m[1mResolving...[22m
[90m===[39m
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Project.toml`
[32m[1m No Changes[22m[39m to `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_odetpikzic/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
[32m[1mPrecompiling[22m[39m packages...
27323.3 ms[32m ✓ [39mZygote
1060.8 ms[32m ✓ [39mZygote → ZygoteColorsExtenabled÷restart_recommended_msgrestart_required_msgbusy_packageswaiting_for_permission,waiting_for_permission_but_probably_disabled«cell_inputs O$cf264b86-d86f-4c4a-af3b-03b34486e5ffcell_id$cf264b86-d86f-4c4a-af3b-03b34486e5ffcodeْmd"### Visualization
Let's test our optimizers. We will simulate a loss function that depends on two parameters using the 2D Rosenbrock function:"metadatashow_logsèdisabled®skip_as_script«code_folded$ac1355a2-db94-4954-ad65-6887b7dbb7adcell_id$ac1355a2-db94-4954-ad65-6887b7dbb7adcodestd(transformed_data)metadatashow_logsèdisabled®skip_as_script«code_folded$3a100e93-7bba-4836-a4e6-e285aa6406c4cell_id$3a100e93-7bba-4836-a4e6-e285aa6406c4codehtest_data = [
-2.463,
-0.934,
-1.854,
-3.875,
2.477,
-1.1875,
-1.158,
-1.224,
-1.344,
-0.319,
-0.525,
-1.388,
-1.837,
-1.316,
-0.597,
-1.068,
0.1412,
-0.959,
-1.479,
-2.004,
-0.1953,
-1.222,
-1.457,
-0.768,
-0.4158,
-3.06,
-1.428,
-1.165,
-0.226,
-1.204,
-1.135,
-0.835,
-2.46,
-1.36,
-1.239,
-1.266,
-0.8613,
-1.226,
-1.055,
-1.037,
-1.969,
-1.564,
-0.982,
-1.238,
-1.414,
-2.246,
-2.925,
-1.3955,
-1.237,
-1.005,
]metadatashow_logsèdisabled®skip_as_script«code_folded$80c5a262-e758-4075-896f-a0bc7ce1534ccell_id$80c5a262-e758-4075-896f-a0bc7ce1534ccode md"Let's see if our code works:"metadatashow_logsèdisabled®skip_as_script«code_folded$8b519a7d-b64f-4c0c-8d74-17a963cb381dcell_id$8b519a7d-b64f-4c0c-8d74-17a963cb381dcodeXtip(md"It is recommended to quickly review exercise 3 from homework 1 before starting.")metadatashow_logsèdisabled®skip_as_script«code_folded$0fbd2560-1e4c-45b7-89a6-d2b622f9f907cell_id$0fbd2560-1e4c-45b7-89a6-d2b622f9f907codefunction plot_loss_surface(; kwargs...)
return contour(
P1,
P2,
(p1, p2) -> loss((p1, p2));
xlabel="Parameter 1",
ylabel="Parameter 2",
kwargs...,
)
endmetadatashow_logsèdisabled®skip_as_script«code_folded$ff5d316f-806c-4652-97d8-323462395c69cell_id$ff5d316f-806c-4652-97d8-323462395c69code٦# Please don't edit your name here, but in the cell below. Thanks!
md"Submission by: **_$(student.name)_** ($(student.email), Matr.-Nr.: $(student.id))" # DO NOT EDITmetadatashow_logsèdisabled®skip_as_script«code_folded$e5e014e6-e24d-4de0-a216-61c0f9d9c331cell_id$e5e014e6-e24d-4de0-a216-61c0f9d9c331codetypeof(dt)metadatashow_logsèdisabled®skip_as_script«code_folded$02a62f7b-6014-439a-a254-49f53f204abbcell_id$02a62f7b-6014-439a-a254-49f53f204abbcode7md"### Exercise 1.1 – Composite types & constructors"metadatashow_logsèdisabled®skip_as_script«code_folded$bcdd297b-c52c-4076-9329-929d69b8021bcell_id$bcdd297b-c52c-4076-9329-929d69b8021bcodemd"### Visualization
As we can see in this plot, our original data didn't come from a normal distribution.
Standardization transforms the data such that $\mu = 0$ and $\sigma=1$, but doesn't make it normally distributed.
"metadatashow_logsèdisabled®skip_as_script«code_folded$3f81ccce-5744-42ba-aa1e-b3da6212b9efcell_id$3f81ccce-5744-42ba-aa1e-b3da6212b9efcode>plot_trajectory(optimizers; steps=50, params_init=[-2.0, 1.0])metadatashow_logsèdisabled®skip_as_script«code_folded$a3634333-1149-4392-a7e9-350a56d67d2bcell_id$a3634333-1149-4392-a7e9-350a56d67d2bcodefunction plot_trajectory(optimizers; steps=30, params_init=[-1.0, 1.0])
# Plot the level sets of the loss function
p = plot_loss_surface(; c=:gray, colorbar=false)
# Allocate an array to save the trajectory
trajectory = similar(params_init, 2, steps + 1)
trajectory[:, 1] = params_init
for (name, opt) in optimizers
# Use initial parameters
params = params_init
# Setup the optimizer
state = Optimisers.setup(opt, params)
for i in 1:steps
# Compute the gradient
# We will learn how this works in the lecture on autodiff next week
grad = Zygote.gradient(loss, params)[1]
# Update state and parameters
state, params = Optimisers.update(state, params, grad)
# Save params into trajectory
trajectory[:, i+1] = params
end
# Plot trajectory on top of the loss surface
plot!(
p,
trajectory[1, :],
trajectory[2, :];
label="$name",
marker=:o,
markerstrokewidth=0.5,
markersize=2,
lw=2,
legend=:topright,
xlims=(-2, 2),
ylims=(-2, 2),
)
end
return p
endmetadatashow_logsèdisabled®skip_as_script«code_folded$be2c5285-fbd8-4c37-8b17-db6db95c8a4ecell_id$be2c5285-fbd8-4c37-8b17-db6db95c8a4ecodetask(
md"Now define two functions `transform` and `inverse_transform`:
1. A function `transform` that takes an array and a `Standardize` object and implements the standardization
$X' = \frac{X-\mu}{\sigma} \quad .$
2. A function `inverse_transform` that takes an array and a `Standardize` object and implements the inverse standardization
$X = \left(X' \cdot \sigma\right) + \mu \quad .$
",
2,
)metadatashow_logsèdisabled®skip_as_script«code_folded$ef9a0669-2d02-4f1b-aa99-6f6473d37c0fcell_id$ef9a0669-2d02-4f1b-aa99-6f6473d37c0fcodeMtask(
md"Define a type `GradientDescent` that implements [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent). It should:
* be a sub-type of `AbstractRule`
* have a learning rate `lr` as its only field
* have a default learning rate of `0.1`, implemented in an outer constructor `GradientDescent()`
",
2,
)metadatashow_logsèdisabled®skip_as_script«code_folded$aaea0bbf-868d-4fe0-b348-307449d06517cell_id$aaea0bbf-868d-4fe0-b348-307449d06517code# Write your code here!metadatashow_logsèdisabled®skip_as_script«code_folded$c080e799-b5c5-42cf-b5f1-8c36da89fc8fcell_id$c080e799-b5c5-42cf-b5f1-8c36da89fc8fcode!md"### Exercise 2.3 – `apply!`"metadatashow_logsèdisabled®skip_as_script«code_folded$569173af-9e54-49cb-b1cb-88b8cbda57a6cell_id$569173af-9e54-49cb-b1cb-88b8cbda57a6codeUmd"And let's test the inverse transformation. The following vectors should be equal:"metadatashow_logsèdisabled®skip_as_script«code_folded$f60be2e0-9b43-46b5-96ef-7747ab56e164cell_id$f60be2e0-9b43-46b5-96ef-7747ab56e164codeٺfeedback = """
The homework took me around XX minutes.
In the lecture / homework I would have liked more detailed explanations on:
* foo
* bar
I liked:
* baz
I didn't like:
* qux
""";metadatashow_logsèdisabled®skip_as_script«code_folded$dc35d701-ab40-4933-b4c5-35a81b3f834acell_id$dc35d701-ab40-4933-b4c5-35a81b3f834acode>@bind lr1 Slider(0.01:0.01:0.2, default=0.15, show_value=true)metadatashow_logsèdisabled®skip_as_script«code_folded$a96abf48-cacf-41f8-bb7a-3d085bdb0a62cell_id$a96abf48-cacf-41f8-bb7a-3d085bdb0a62codeLmd"You can also implement [more variants](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Extensions_and_variants) or simply use [those implemented in Optimisers.jl](https://fluxml.ai/Optimisers.jl/dev/api/#Optimisation-Rules).
Click `plot_trajectory` below to reveal the code that produced the plot.
The most relevant part is the inner loop where the optimizer is called:
```julia
state = Optimisers.setup(opt, params)
for i in 1:steps
grad = ... # Compute gradient. We will learn how this works next week.
state, params = Optimisers.update(state, params, grad)
end
```
"metadatashow_logsèdisabled®skip_as_script«code_folded$58a671a1-3059-4fa9-b758-f3a76a573aa5cell_id$58a671a1-3059-4fa9-b758-f3a76a573aa5code# Write your code heremetadatashow_logsèdisabled®skip_as_script«code_folded$4eac0e3f-7212-4038-a24e-8d6ca3fbe659cell_id$4eac0e3f-7212-4038-a24e-8d6ca3fbe659codeNif !@isdefined(GradientDescent)
keep_working(md"Make sure that you define a struct called `GradientDescent`.")
elseif !hasmethod(
Optimisers.apply!, Tuple{GradientDescent,Nothing,Vector{Float64},Vector{Float64}}
)
keep_working()
else
let
gd = GradientDescent(2.0)
grad = [1.0, 2.0]
state_new, grad_new = Optimisers.apply!(gd, nothing, [0.0, 0.0], grad)
if ismissing(grad_new)
still_missing()
elseif isnothing(grad_new)
keep_working()
elseif !isnothing(state_new)
keep_working(md"Don't modify the state.")
elseif !(grad_new ≈ [2.0, 4.0])
keep_working(
md"The answer is not quite right. There is an error in the update of the gradient.",
)
else
correct()
end
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$3d638561-6760-4260-991a-b0b0c77745a3cell_id$3d638561-6760-4260-991a-b0b0c77745a3codeusing Statisticsmetadatashow_logsèdisabled®skip_as_script«code_folded$17ff11e9-84f8-4911-845b-930a53c73e0ecell_id$17ff11e9-84f8-4911-845b-930a53c73e0ecodedt.μmetadatashow_logsèdisabled®skip_as_script«code_folded$3ac8faba-1af9-4fc6-a1d0-37225b9ebb02cell_id$3ac8faba-1af9-4fc6-a1d0-37225b9ebb02code]md"## Exercise 2 – Gradient-based optimization
As we have seen in the previous exercise, strucs / composite types are a common way to implement algorithms.
In this exercise, we will extend the [gradient-based optimization algorithms](https://en.wikipedia.org/wiki/Gradient_descent) from [Optimisers.jl](https://github.com/FluxML/Optimisers.jl/).
"metadatashow_logsèdisabled®skip_as_script«code_folded$bb0d58f9-380e-43fa-9524-0108a2816884cell_id$bb0d58f9-380e-43fa-9524-0108a2816884codebfunction transform(xs, dt::Standardize)
return missing # Replace `missing` with your code!
endmetadatashow_logsèdisabled®skip_as_script«code_folded$5061a130-fc0a-4306-bdf5-6966e8de938acell_id$5061a130-fc0a-4306-bdf5-6966e8de938acodexbegin
md_parse(str) = Markdown.parse(str)
md_parse(md::Markdown.MD) = md
function task(text, pts=1)
if pts == 1
pts_str = "(1 point)"
elseif pts == 0
pts_str = ""
else
pts_str = "($pts points)"
end
Markdown.MD(Markdown.Admonition("Task", "Task " * pts_str, [md_parse(text)]))
end
end;metadatashow_logsèdisabled®skip_as_script«code_folded$ddd6e83e-5a0d-4ff0-afe4-dedfc860994ccell_id$ddd6e83e-5a0d-4ff0-afe4-dedfc860994ccodemd"### Student information"metadatashow_logsèdisabled®skip_as_script«code_folded$f9192279-9d12-485e-bb9d-650be2dbd878cell_id$f9192279-9d12-485e-bb9d-650be2dbd878codeA@bind lr2 Slider(0.001:0.001:0.05, default=0.01, show_value=true)metadatashow_logsèdisabled®skip_as_script«code_folded$78fcd02f-1a6e-4799-b042-4542a1ec7847cell_id$78fcd02f-1a6e-4799-b042-4542a1ec7847code#plot_standardization(test_data, dt)metadatashow_logsèdisabled®skip_as_script«code_folded$2341dd4a-a35a-4a62-815a-e972d04844abcell_id$2341dd4a-a35a-4a62-815a-e972d04844abcodeٹmd"""Let's plot the trajectory of our optimizers on the "loss surface".
Add as many optimizers and sliders as you want, change the initial parameters and the number of steps taken.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$d36914f8-f7d4-407a-a842-436f4035157bcell_id$d36914f8-f7d4-407a-a842-436f4035157bcodeٰmd"## Exercise 1 – Standardization
In this exercise, we will refactor the standardization code from homework 1 by using custom types.
Let's load the dataset from last time:
"metadatashow_logsèdisabled®skip_as_script«code_folded$3b65f124-75c2-4fb3-8b59-94015500acbfcell_id$3b65f124-75c2-4fb3-8b59-94015500acbfcode"P1 = P2 = range(-2, 2; length=51);metadatashow_logsèdisabled®skip_as_script«code_folded$755b8685-0711-48a2-a3eb-f80af39f10e1cell_id$755b8685-0711-48a2-a3eb-f80af39f10e1code٣begin
using PlutoUI
using PlutoTeachingTools
using Zygote
using LaTeXStrings
using Plots
import PlotlyBase, PlotlyKaleido
plotly()
endmetadatashow_logsèdisabled®skip_as_script«code_folded$3bb76227-82cd-4924-bfb3-1b8504e2dc44cell_id$3bb76227-82cd-4924-bfb3-1b8504e2dc44codeJloss(params) = 2 * params[1]^2 + (2 * params[2] - params[1])^2 - params[1]metadatashow_logsèdisabled®skip_as_script«code_folded$5dc7d04d-f6a5-4265-b890-d74ea4b3c602cell_id$5dc7d04d-f6a5-4265-b890-d74ea4b3c602code# Write your code heremetadatashow_logsèdisabled®skip_as_script«code_folded$fd30da51-0890-4ebe-8b14-8975ec12fec0cell_id$fd30da51-0890-4ebe-8b14-8975ec12fec0codeif !@isdefined(Standardize)
keep_working(md"Make sure that you define a struct called `Standardize`.")
elseif !hasmethod(Standardize, Tuple{Int,Int})
keep_working(md"The inner constructor has not yet been defined.")
elseif !hasmethod(Standardize, Tuple{Vector{Int}})
keep_working(md"The outer constructor has not yet been defined.")
else
let
dtt1 = Standardize(0, 1)
train = [1, 4, 5]
dtt2 = Standardize(train)
if ismissing(dtt1)
still_missing()
elseif isnothing(dtt1)
keep_working(
md"Did you forget to return a `new` object in the inner constructor?"
)
elseif ismissing(dtt1)
still_missing(md"Your inner constructor still returns `missing`.")
elseif isnothing(dtt2)
keep_working(md"Did you forget to write `return` in the outer constructor?")
elseif ismissing(dtt2)
still_missing(md"Replace `missing` in the outer constructor with your answer.")
elseif !(dtt2.μ ≈ 3.3333333333333335)
keep_working(
md"The answer is not quite right. The mean `μ` wasn't computed correctly in the outer constructor.",
)
elseif !(dtt2.σ ≈ 2.081665999466133)
keep_working(
md"The answer is not quite right. The standard deviation `σ` wasn't computed correctly in the outer constructor.",
)
else
correct()
end
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$8ece9aea-20f5-41db-95ca-08c8d4d2d4c1cell_id$8ece9aea-20f5-41db-95ca-08c8d4d2d4c1codeAtask("Please add your student information to the cell below.", 0)metadatashow_logsèdisabled®skip_as_script«code_folded$b0d679ba-13f8-4e70-ac42-68acb2ff83d1cell_id$b0d679ba-13f8-4e70-ac42-68acb2ff83d1code١task(md"Define an `Optimisers.init` function for `GradientDescent` that returns `nothing`.
We return `nothing` since gradient descent doesn't require a state.")metadatashow_logsèdisabled®skip_as_script«code_folded$f83196a4-a0fd-44a3-b3d1-29730599830dcell_id$f83196a4-a0fd-44a3-b3d1-29730599830dcode+transformed_data = transform(test_data, dt)metadatashow_logsèdisabled®skip_as_script«code_folded$6d5bf1e5-5ab0-4351-89d0-ddb6fcfa27b6cell_id$6d5bf1e5-5ab0-4351-89d0-ddb6fcfa27b6code if !@isdefined(GradientDescent)
keep_working(md"Make sure that you define a struct called `GradientDescent`.")
elseif !hasmethod(GradientDescent, Tuple{Float64})
keep_working(md"The inner constructor has not yet been defined.")
elseif !hasmethod(GradientDescent, Tuple{})
keep_working(md"The outer constructor has not yet been defined.")
else
let
gd1 = GradientDescent(1.2)
gd2 = GradientDescent()
if !hasfield(GradientDescent, :lr)
keep_working(md"`GradientDescent` should have a field called `lr`.")
elseif !(gd1.lr ≈ 1.2)
keep_working()
elseif !(gd2.lr ≈ 0.1)
keep_working(md"The default value for `lr` should be `0.1`.")
else
correct()
end
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$822d03c6-48f8-47b0-8f25-b2de45e4c589cell_id$822d03c6-48f8-47b0-8f25-b2de45e4c589codeeMarkdown.MD(
Markdown.Admonition(
"Task",
"Optional task",
[
md"Implement the *Adam* optimizer, which was introduced
in [this paper](https://arxiv.org/abs/1412.6980) by Kingma and Ba.
Add it to the visualization above and compare it to Optimizer.jl's Adam implementation.",
],
),
)metadatashow_logsèdisabled®skip_as_script«code_folded$bebbf87b-cd7b-43d7-8025-938fc4c6944dcell_id$bebbf87b-cd7b-43d7-8025-938fc4c6944dcodetask(md"Define an `Optimisers.apply!` function for `GradientDescent` that
* doesn't modify the state (since gradient descent doesn't require a state)
* scales the gradient by the learning rate
")metadatashow_logsèdisabled®skip_as_script«code_folded$c2adc38f-84e3-486f-93d8-b88eea63a09bcell_id$c2adc38f-84e3-486f-93d8-b88eea63a09bcode6md"### Exercise 1.2 – Transform & Inverse transform"metadatashow_logsèdisabled®skip_as_script«code_folded$5370c50b-a9b8-41dc-8cfe-1f0389df97c3cell_id$5370c50b-a9b8-41dc-8cfe-1f0389df97c3code(md"### Optional Task – Adam optimizer"metadatashow_logsèdisabled®skip_as_script«code_folded$32e8ee04-4378-4377-ae47-b65e5ce70667cell_id$32e8ee04-4378-4377-ae47-b65e5ce70667codemd"### Exercise 2.2 – `init`"metadatashow_logsèdisabled®skip_as_script«code_folded$befba318-8595-4fc4-b42e-9c9125e7abfbcell_id$befba318-8595-4fc4-b42e-9c9125e7abfbcodeDtip(
md"Statefulness could also have been implemented by making the state a field in the struct of the optimizer and/or making the struct mutable.
However, mutable structs are usually less performant. The dimensionality of the state also often depends on the dimensionality of the input parameters we optimize over.",
)metadatashow_logsèdisabled®skip_as_script«code_folded$d5052369-0b47-42e7-b661-058c7365c032cell_id$d5052369-0b47-42e7-b661-058c7365c032codeٕtip(
md"**Reminder:** By convention, functions that modify their inputs in-place are given an exclamation point `!` at the end of their name.",
)metadatashow_logsèdisabled®skip_as_script«code_folded$cf3c52c7-fc5e-4191-be9d-394660385801cell_id$cf3c52c7-fc5e-4191-be9d-394660385801code;dt = Standardize(test_data) # dt like "data transformation"metadatashow_logsèdisabled®skip_as_script«code_folded$2cd9cef0-d746-411a-861f-b8a035f4a1d3cell_id$2cd9cef0-d746-411a-861f-b8a035f4a1d3codemean(transformed_data)metadatashow_logsèdisabled®skip_as_script«code_folded$d15eb189-6e4d-4c70-a1cb-159e6a4eb8ebcell_id$d15eb189-6e4d-4c70-a1cb-159e6a4eb8ebcodetip(
md"Gradient descent is typically used in deep learning in combination with mini-batches, which introduce stochasticity.
This example had no stochasticity and other optimization algorithms would have been better suited.",
)metadatashow_logsèdisabled®skip_as_script«code_folded$955a9751-38d0-4327-9a95-a0523c2d54b8cell_id$955a9751-38d0-4327-9a95-a0523c2d54b8codedt.σmetadatashow_logsèdisabled®skip_as_script«code_folded$62a1289b-ce6a-444d-8b20-92bd4fa3d05dcell_id$62a1289b-ce6a-444d-8b20-92bd4fa3d05dcodeAmd"For the purpose of plotting, we define a range of parameters:"metadatashow_logsèdisabled®skip_as_script«code_folded$221fd504-4cd6-4bab-b2a5-a590fff26a52cell_id$221fd504-4cd6-4bab-b2a5-a590fff26a52codejfunction inverse_transform(xs, dt::Standardize)
return missing # Replace `missing` with your code!
endmetadatashow_logsèdisabled®skip_as_script«code_folded$a191392f-eb62-459d-8554-901d9db29950cell_id$a191392f-eb62-459d-8554-901d9db29950code5inverse_transform(transformed_data, dt) ≈ test_datametadatashow_logsèdisabled®skip_as_script«code_folded$67d7c60c-1c5e-4760-a9e1-4538e2ccc10ecell_id$67d7c60c-1c5e-4760-a9e1-4538e2ccc10ecodemd"""#### Initializing the optimizer
Some optimizers require a state. This state is used to save additional information about previous optimization steps, for example in optimizers that use "momentum" terms like [ADAM](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Adam).
When setting up an Optimizer in Optimisers.jl, `init` is called on it, which initializes a state.
`init` takes two arguments: an optimizer object and an array of initial parameters.
```julia
function Optimisers.init(alg::MyOptimiser, params_init::AbstractArray)
state = ... # some computation
return state
end
```
The state is then updated in the `apply!` function.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$d16b1a65-5da9-46ac-b0b8-59e55ced37f3cell_id$d16b1a65-5da9-46ac-b0b8-59e55ced37f3code$md"### Exercise 2.1 – Constructor"metadatashow_logsèdisabled®skip_as_script«code_folded$7de85c6f-996a-4d5b-8236-271922aeb908cell_id$7de85c6f-996a-4d5b-8236-271922aeb908codefunction plot_standardization(xs, dt::Standardize; nbins=20, scale=80)
p = histogram(xs; nbins=nbins, alpha=0.7, label="Original data")
histogram!(p, transform(xs, dt); nbins=nbins, alpha=0.5, label="Standardized data")
plot!(
p,
x -> 1 / (2π) * exp(-0.5 * x^2) * scale;
color=:gray,
lw=2,
style=:dash,
label="Standard normal distribution (scaled)",
)
return p
endmetadatashow_logsèdisabled®skip_as_script«code_folded$edb7814a-eddf-4c87-8857-19bb0a0c0241cell_id$edb7814a-eddf-4c87-8857-19bb0a0c0241codemd"""# Feedback
This is the fourth iteration of the *"Julia for Machine Learning"* class. Please help us make the course better!
You can write whatever you want in the following string. Feel free to add or delete whatever you want.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$2258b8c6-e8b7-4c06-9ece-53b254db6a2ccell_id$2258b8c6-e8b7-4c06-9ece-53b254db6a2ccodetask(
md"Create a parametric composite type `Standardize` with two fields `μ` and `σ` using the type parameter `T<:Real`.
These two fields will hold the mean and the standard deviation of the standardization transformation.
`Standardize` should have two constructors:
1. An inner constructor that takes two numbers $μ$ and $σ$. Throw an `error` if $σ ≤ 0$.
2. An outer constructor that takes an array `data`, computes its mean and standard deviation and calls the inner constructor.
---
You are allowed to use the `mean` and `std` functions from the standard library package [Statistics.jl](https://docs.julialang.org/en/v1/stdlib/Statistics/).
",
4,
)metadatashow_logsèdisabled®skip_as_script«code_folded$8e8d9f57-0f04-4a91-b69f-273fe1fafd61cell_id$8e8d9f57-0f04-4a91-b69f-273fe1fafd61code5html""""""metadatashow_logsèdisabled®skip_as_script«code_folded$f7347c06-c1b7-11ed-3b8e-fbf167ce9cbacell_id$f7347c06-c1b7-11ed-3b8e-fbf167ce9cbacode
Julia for Machine Learning
Homework 3: Custom types
TU Berlin, Winter Semester 25/26
"""metadatashow_logsèdisabled®skip_as_script«code_folded$c322fd53-78db-40ae-b7ba-612200ab6617cell_id$c322fd53-78db-40ae-b7ba-612200ab6617codepmd"""### Interlude: Gradient-based optimization in Deep Learning
Gradient-based optimization methods are used in deep learning, where we want to optimize a neural network $f$ that is parametrized by parameters $\theta$ (e.g. the weights and biases of $f$).
This neural network could e.g. be a classifier that predicts the output class $\hat{y}$ for some input image $x$:
$\hat{y} = f_\theta(x)$
In [supervised learning](https://en.wikipedia.org/wiki/Supervised_learning),
we are also given true labels $y$.
Based on the true and predicted labels, we define a loss
$\mathcal{L}_\theta(\hat{y}, y)$
which depends on $\theta$, since the parameters of our neural network determine the prediction $\hat{y}$.
To optimize $\theta$, we typically compute the gradient of our loss with respect to our parameters and update the parameters using gradient-based optimization methods.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$dc4f7f55-dde5-4ee6-b43d-7f64269b55d1cell_id$dc4f7f55-dde5-4ee6-b43d-7f64269b55d1code# Write your code heremetadatashow_logsèdisabled®skip_as_script«code_folded$cd7c244d-8a4c-4aaf-ad4b-891744e004becell_id$cd7c244d-8a4c-4aaf-ad4b-891744e004becodeplot_loss_surface(; fill=true)metadatashow_logsèdisabled®skip_as_script«code_folded$ed873ea2-d4bd-46e7-b89b-4e5494b606e6cell_id$ed873ea2-d4bd-46e7-b89b-4e5494b606e6codeoptimizers = Dict(
"GradientDescent" => GradientDescent(lr1), # our implementation
"Momentum" => Momentum(lr2), # from Optimisers.jl
"ADAM" => ADAM(0.1), # from Optimisers.jl
)metadatashow_logsèdisabled®skip_as_script«code_folded$b3832fb6-dd19-4b38-8117-b0a11d1bc34fcell_id$b3832fb6-dd19-4b38-8117-b0a11d1bc34fcode md"""### Interlude: Understanding Optimisers.jl
All optimization algorithms in Optimisers.jl are sub-types of the abstract type `AbstractRule`.
To extend Optimisers.jl, we simply need to define a custom sub-type and overload two functions `init` and `apply!` using multiple dispatch.
"""metadatashow_logsèdisabled®skip_as_script«code_folded$bdcb27c5-0603-49ac-b831-d78c558b31f0cell_id$bdcb27c5-0603-49ac-b831-d78c558b31f0code5md"Due date: **Monday, November 10th 2025 at 23:59**"metadatashow_logsèdisabled®skip_as_script«code_folded$33866006-11d7-46cc-870b-fec519a2b541cell_id$33866006-11d7-46cc-870b-fec519a2b541codevtip(
md"The purpose of this exercise is to teach you how to extend functionality from external Julia packages.",
)metadatashow_logsèdisabled®skip_as_script«code_folded$44ec9e94-f6af-431e-8841-bae44431dfa3cell_id$44ec9e94-f6af-431e-8841-bae44431dfa3codeif student.name == "Mara Mustermann"
still_missing(md"""Please replace `"Mara Mustermann"` with your name.
Use `Shift+Enter` to run your edits.""")
elseif student.email == "m.mustermann@campus.tu-berlin.de"
still_missing(md"Please enter your TU Berlin E-Mail address.")
elseif student.id == 456123
still_missing(md"Please enter your TU Berlin student ID (Matrikelnummer).")
elseif !isa(student.id, Number)
still_missing(md"Please enter your TU Berlin student ID as number, not a string.")
else
correct()
endmetadatashow_logsèdisabled®skip_as_script«code_folded$57a8108f-9ad3-4cc2-9372-bb291fc3bf86cell_id$57a8108f-9ad3-4cc2-9372-bb291fc3bf86codeTableOfContents()metadatashow_logsèdisabled®skip_as_script«code_folded$50b34968-a45f-4c5b-a83d-7a5b47740007cell_id$50b34968-a45f-4c5b-a83d-7a5b47740007codemd"""#### Applying the update step
The actual update step is handled automatically by Optimisers.jl.
After implementing the "internal" functions `init` and `apply!` to define a custom optimizer, we can use the "public" API:
- `Optimisers.setup`: calls `init` to set up the optimizer
- `Optimisers.update`: calls `apply!` to compute `new_grad` and `next_state` and performs the update step `params_new = params - new_grad`
"""metadatashow_logsèdisabled®skip_as_script«code_folded$06c0ad65-22d4-4c8e-ae19-4f05ba125e79cell_id$06c0ad65-22d4-4c8e-ae19-4f05ba125e79codekmd"### Initializing packages
**Note:** Running this notebook for the first time can take several minutes.
"metadatashow_logsèdisabled®skip_as_script«code_folded$dcbf8963-4a5c-4e7d-81a9-3dce27fb3117cell_id$dcbf8963-4a5c-4e7d-81a9-3dce27fb3117codemd"Let's check whether the transformed data has $\mu'=0$ and $\sigma'=1$.
Note that due to [rounding errors](https://en.wikipedia.org/wiki/Machine_epsilon) arising from floating point arithmetic, the results will not be exactly zero and one:
"metadatashow_logsèdisabled®skip_as_script«code_folded$5762be30-3084-4eda-9642-19dc9d4dcb10cell_id$5762be30-3084-4eda-9642-19dc9d4dcb10codeif !@isdefined(GradientDescent)
keep_working(md"Make sure that you define a struct called `GradientDescent`.")
elseif !hasmethod(Optimisers.init, Tuple{GradientDescent,Vector{Float64}})
keep_working()
else
let
gd = GradientDescent()
state = Optimisers.init(gd, [1.0])
if !isnothing(state)
keep_working(md"Initializing `GradientDescent` should return `nothing`.")
else
correct()
end
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$39063d08-d687-45a2-8367-734b12dcefeacell_id$39063d08-d687-45a2-8367-734b12dcefeacode(if !@isdefined(Standardize)
keep_working(md"Make sure that you define a struct called `Standardize`.")
elseif !@isdefined(transform)
not_defined(:transform)
elseif !@isdefined(inverse_transform)
not_defined(:inverse_transform)
else
let
train = [1, 4, 5]
test = [2, 1, 3]
dtt = Standardize(train)
res_tf = transform(test, dtt)
sol_tf = [-0.6405126152203485, -1.12089707663561, -0.1601281538050872]
res_itf = inverse_transform(res_tf, dtt)
if ismissing(res_tf)
still_missing(md"Replace `missing` in `transform` with your answer.")
elseif isnothing(res_tf)
keep_working(md"Did you forget to write `return` in `transform`?")
elseif !(res_tf ≈ sol_tf)
keep_working(md"The answer is not quite right. Keep working on `transform`.")
elseif ismissing(res_itf)
still_missing(md"Replace `missing` in `inverse_transform` with your answer.")
elseif isnothing(res_itf)
keep_working(md"Did you forget to write `return` in `inverse_transform`?")
elseif !(res_itf ≈ test)
keep_working(
md"The answer is not quite right. Keep working on `inverse_transform`."
)
else
correct()
end
end
endmetadatashow_logsèdisabled®skip_as_script«code_folded$582cf667-b85b-4a7d-bb17-88d4776fc240cell_id$582cf667-b85b-4a7d-bb17-88d4776fc240codeusing Optimisersmetadatashow_logsèdisabled®skip_as_script«code_folded$d03e4e95-faab-4ab3-ab27-81189cbd8231cell_id$d03e4e95-faab-4ab3-ab27-81189cbd8231codeّstudent = (
name="Mara Mustermann",
email="m.mustermann@campus.tu-berlin.de", # TU Berlin email address
id=456123, # Matrikelnummer
)metadatashow_logsèdisabled®skip_as_script«code_folded$e0705a25-a01e-43a8-8c1d-ccf0da22fca7cell_id$e0705a25-a01e-43a8-8c1d-ccf0da22fca7codeumd"""#### Updating the optimizer
The function `apply!` is can do two things before the final update step is applied:
1. it can modify the gradient
1. it can update the optimizer state
```julia
function Optimisers.apply!(alg::MyOptimiser, state, params, grad)
grad_new = ... # some computation
state_new = ... # some computation
return state_new, grad_new
end
```
"""metadatashow_logsèdisabled®skip_as_script«code_foldedënotebook_id$ac42f60e-a829-11f0-147b-db1b68dd41adin_temp_dir¨metadata